[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:28:35 dist-test-slave-dssx krb5kdc[6057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:28:35 dist-test-slave-dssx krb5kdc[6057](info): set up 2 sockets
Sep 29 16:28:35 dist-test-slave-dssx krb5kdc[6057](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 16:28:36.010175 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.009s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:28:36 dist-test-slave-dssx krb5kdc[6057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:28:36.191740 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:37111
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:37111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:36.544581 6070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:36.544910 6070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:36.545002 6070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:36.552990 6070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:36.553119 6070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:36.553167 6070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:36.553225 6070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:36.553270 6070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:36.563446 6070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:37111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:37111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:36.564885 6070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:36.566360 6070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:36.579304 6075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:36.579327 6076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:36.579291 6078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:36.580093 6070 server_base.cc:1034] running on GCE node
I20260929 16:28:36.580950 6070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:36.582558 6070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:36.583815 6070 hybrid_clock.cc:648] HybridClock initialized: now 1790699316583751 us; error 63 us; skew 500 ppm
I20260929 16:28:36.584178 6070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:36 dist-test-slave-dssx krb5kdc[6057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699316, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:36.587417 6070 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:28:36.589663 6070 webserver.cc:460] Webserver started at http://127.5.232.190:45425/ using document root <none> and password file <none>
I20260929 16:28:36.590436 6070 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:36.590600 6070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:36.591048 6070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:36.594362 6070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "2cbf9161eb8c4c32b89f37e9379e360d"
format_stamp: "Formatted at 2026-09-29 16:28:36 on dist-test-slave-dssx"
server_key: "d90d01dbee49c254d714e64e0e8fb672"
server_key_iv: "93918112c48bb2d8111b7be4e1481fce"
server_key_version: "encryptionkey@0"
I20260929 16:28:36.595208 6070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "2cbf9161eb8c4c32b89f37e9379e360d"
format_stamp: "Formatted at 2026-09-29 16:28:36 on dist-test-slave-dssx"
server_key: "d90d01dbee49c254d714e64e0e8fb672"
server_key_iv: "93918112c48bb2d8111b7be4e1481fce"
server_key_version: "encryptionkey@0"
I20260929 16:28:36.600827 6070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:28:36.604682 6085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:36.606205 6070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:36.606418 6070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "2cbf9161eb8c4c32b89f37e9379e360d"
format_stamp: "Formatted at 2026-09-29 16:28:36 on dist-test-slave-dssx"
server_key: "d90d01dbee49c254d714e64e0e8fb672"
server_key_iv: "93918112c48bb2d8111b7be4e1481fce"
server_key_version: "encryptionkey@0"
I20260929 16:28:36.606637 6070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:36.627737 6070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:36.632524 6070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:36.632829 6070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:36.652921 6070 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:37111
I20260929 16:28:36.652930 6136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:37111 every 8 connection(s)
I20260929 16:28:36.654559 6070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:36.659044 6137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:36.660679 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 6070
I20260929 16:28:36.661051 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:28:36.663085 6050 external_mini_cluster.cc:1428] Setting key f3272bf1c463e87efd3ecc6424a59c58
I20260929 16:28:36.672149 6137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d: Bootstrap starting.
I20260929 16:28:36.675535 6137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:36.676662 6137 log.cc:824] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:36.679968 6137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d: No bootstrap required, opened a new log
I20260929 16:28:36.688073 6137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbf9161eb8c4c32b89f37e9379e360d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37111 } }
I20260929 16:28:36.688555 6137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:36.688683 6137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf9161eb8c4c32b89f37e9379e360d, State: Initialized, Role: FOLLOWER
I20260929 16:28:36.689313 6137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbf9161eb8c4c32b89f37e9379e360d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37111 } }
I20260929 16:28:36.689661 6137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:36.689800 6137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:36.689998 6137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:36.692346 6137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbf9161eb8c4c32b89f37e9379e360d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37111 } }
I20260929 16:28:36.693008 6137 leader_election.cc:302] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cbf9161eb8c4c32b89f37e9379e360d; no voters:
Sep 29 16:28:36 dist-test-slave-dssx krb5kdc[6057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:36.693826 6137 leader_election.cc:288] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:28:36.693935 6142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:36.695226 6142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [term 1 LEADER]: Becoming Leader. State: Replica: 2cbf9161eb8c4c32b89f37e9379e360d, State: Running, Role: LEADER
I20260929 16:28:36.695943 6142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbf9161eb8c4c32b89f37e9379e360d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37111 } }
I20260929 16:28:36.696404 6137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:36.699024 6144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cbf9161eb8c4c32b89f37e9379e360d. Latest consensus state: current_term: 1 leader_uuid: "2cbf9161eb8c4c32b89f37e9379e360d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbf9161eb8c4c32b89f37e9379e360d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37111 } } }
I20260929 16:28:36.699553 6144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:36.700213 6143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cbf9161eb8c4c32b89f37e9379e360d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbf9161eb8c4c32b89f37e9379e360d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37111 } } }
I20260929 16:28:36.700981 6143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:36.711442 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:36.667415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.5.232.190:37111)
0929 16:28:36.668600 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:36.668627 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:36.669896 (+ 1269us) server_negotiation.cc:407] Connection header received
0929 16:28:36.679138 (+ 9242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:36.679163 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:36.679606 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:36.680064 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:36.683611 (+ 3547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:36.685029 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:36.686959 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:36.687709 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:36.687957 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:36.695258 (+ 7301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:36.695276 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:36.695304 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:36.695371 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:36.699986 (+ 4615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:36.701989 (+ 2003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:36.702004 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:36.702015 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:36.702296 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:36.703114 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:36.703123 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:36.703129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:36.704092 (+ 963us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:36.704528 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:36.708670 (+ 4142us) server_negotiation.cc:299] Negotiation successful
0929 16:28:36.709335 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":623,"threads_started":1}
I20260929 16:28:36.715029 6149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:36.720512 6149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:36.730063 6149 catalog_manager.cc:1347] Generated new cluster ID: fe38a7d63e1b45d5af5ec13d129e5efd
I20260929 16:28:36.730595 6149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:36.750090 6149 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:28:36.752089 6149 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:36.762349 6149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cbf9161eb8c4c32b89f37e9379e360d: Generated new TSK 0
I20260929 16:28:36.763490 6149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:28:36.809157 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37111
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:37.169059 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:37.169364 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:37.169571 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:37.176954 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:37.177076 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:37.177196 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:28:37.187999 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:37.189322 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:37.190982 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:37.205976 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:37.205924 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:37.205901 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:37.210983 6163 server_base.cc:1034] running on GCE node
I20260929 16:28:37.211865 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:37.213023 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:37.214290 6163 hybrid_clock.cc:648] HybridClock initialized: now 1790699317214232 us; error 59 us; skew 500 ppm
I20260929 16:28:37.214650 6163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:37 dist-test-slave-dssx krb5kdc[6057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:37.221894 6163 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:28:37.223592 6163 webserver.cc:460] Webserver started at http://127.5.232.129:41801/ using document root <none> and password file <none>
I20260929 16:28:37.224398 6163 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:37.224560 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:37.224916 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:37.228200 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "98cda38ac87b4894a66ddce8900b557c"
format_stamp: "Formatted at 2026-09-29 16:28:37 on dist-test-slave-dssx"
server_key: "56af7508091af81dc821fe6fc402fb24"
server_key_iv: "29a85e1b969b5bcf8f0d114bc2ebfc32"
server_key_version: "encryptionkey@0"
I20260929 16:28:37.229090 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "98cda38ac87b4894a66ddce8900b557c"
format_stamp: "Formatted at 2026-09-29 16:28:37 on dist-test-slave-dssx"
server_key: "56af7508091af81dc821fe6fc402fb24"
server_key_iv: "29a85e1b969b5bcf8f0d114bc2ebfc32"
server_key_version: "encryptionkey@0"
I20260929 16:28:37.235620 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260929 16:28:37.239579 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:37.241107 6163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:28:37.241319 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "98cda38ac87b4894a66ddce8900b557c"
format_stamp: "Formatted at 2026-09-29 16:28:37 on dist-test-slave-dssx"
server_key: "56af7508091af81dc821fe6fc402fb24"
server_key_iv: "29a85e1b969b5bcf8f0d114bc2ebfc32"
server_key_version: "encryptionkey@0"
I20260929 16:28:37.241626 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:37.268862 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:37.273824 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:37.274144 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:37.275161 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:37.276588 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:37.276705 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:37.276870 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:37.276947 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:37.315856 6163 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:34519
I20260929 16:28:37.315896 6290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:34519 every 8 connection(s)
I20260929 16:28:37.317261 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:28:37.318921 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 6163
I20260929 16:28:37.319164 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:28:37.319876 6050 external_mini_cluster.cc:1428] Setting key 7c855f222330d237e20bd445ee28d10e
Sep 29 16:28:37 dist-test-slave-dssx krb5kdc[6057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
I20260929 16:28:37.342092 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.319996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:43797 (local address 127.5.232.190:37111)
0929 16:28:37.320598 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:37.320605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:37.323941 (+ 3336us) server_negotiation.cc:407] Connection header received
0929 16:28:37.323997 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:37.324001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:37.324065 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:37.324437 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:37.326909 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.327621 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.329371 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.330210 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.330389 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.334278 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:37.334289 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:37.334297 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:37.334351 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:37.338540 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:37.339481 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:37.339486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:37.339488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:37.339608 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:37.340139 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:37.340142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:37.340144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:37.340414 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:37.340523 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:37.341522 (+ 999us) server_negotiation.cc:299] Negotiation successful
0929 16:28:37.341818 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
I20260929 16:28:37.343606 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.320683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:37111 (local address 127.5.232.129:43797)
0929 16:28:37.321652 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:37.321738 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:37.323600 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:37.324797 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:37.324812 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:37.325727 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:37.326662 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.326694 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.327796 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.327801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.329149 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.329168 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.330468 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.330473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.330688 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.331472 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:37.331495 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:37.333959 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:37.338740 (+ 4781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:37.338749 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:37.338767 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:37.339333 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:37.339712 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:37.339715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:37.339718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:37.340024 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:37.340525 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:37.340531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:37.341016 (+ 485us) client_negotiation.cc:769] Sending connection context
0929 16:28:37.341355 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 16:28:37.341980 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":381,"threads_started":1}
I20260929 16:28:37.345602 6291 heartbeater.cc:347] Connected to a master server at 127.5.232.190:37111
I20260929 16:28:37.345958 6291 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:37.346874 6291 heartbeater.cc:511] Master 127.5.232.190:37111 requested a full tablet report, sending...
I20260929 16:28:37.349164 6102 ts_manager.cc:194] Registered new tserver with Master: 98cda38ac87b4894a66ddce8900b557c (127.5.232.129:34519)
I20260929 16:28:37.350749 6102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:43797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:28:37.365649 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37111
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:37.713358 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:37.713785 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:37.713955 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:37.722683 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:37.723009 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:37.723209 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:28:37.735479 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:37.736951 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:37.738478 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:37.753050 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:37.753008 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:37.752985 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:37.753888 6297 server_base.cc:1034] running on GCE node
I20260929 16:28:37.754560 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:37.755535 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:37.756763 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790699317756721 us; error 39 us; skew 500 ppm
I20260929 16:28:37.757076 6297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:37 dist-test-slave-dssx krb5kdc[6057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:37.760310 6297 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:28:37.762347 6297 webserver.cc:460] Webserver started at http://127.5.232.130:44131/ using document root <none> and password file <none>
I20260929 16:28:37.763245 6297 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:37.763360 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:37.763670 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:37.766776 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "eba26d1729af40969f7138f09eb812b2"
format_stamp: "Formatted at 2026-09-29 16:28:37 on dist-test-slave-dssx"
server_key: "74eeb135c5b765175f6f4d949f9a64ea"
server_key_iv: "e50b772c0e894cc225775206add24bee"
server_key_version: "encryptionkey@0"
I20260929 16:28:37.767553 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "eba26d1729af40969f7138f09eb812b2"
format_stamp: "Formatted at 2026-09-29 16:28:37 on dist-test-slave-dssx"
server_key: "74eeb135c5b765175f6f4d949f9a64ea"
server_key_iv: "e50b772c0e894cc225775206add24bee"
server_key_version: "encryptionkey@0"
I20260929 16:28:37.773371 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 16:28:37.777202 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:37.778798 6297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:37.778967 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "eba26d1729af40969f7138f09eb812b2"
format_stamp: "Formatted at 2026-09-29 16:28:37 on dist-test-slave-dssx"
server_key: "74eeb135c5b765175f6f4d949f9a64ea"
server_key_iv: "e50b772c0e894cc225775206add24bee"
server_key_version: "encryptionkey@0"
I20260929 16:28:37.779160 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:37.793288 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:37.798148 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:37.798404 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:37.799346 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:37.800730 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:37.800808 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:37.800892 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:37.800948 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:37.838867 6297 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45199
I20260929 16:28:37.838907 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45199 every 8 connection(s)
I20260929 16:28:37.840210 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:28:37.844435 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 6297
I20260929 16:28:37.844672 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:28:37.845186 6050 external_mini_cluster.cc:1428] Setting key 5ec49b1fef9d4f3d754567beb5b04ec0
Sep 29 16:28:37 dist-test-slave-dssx krb5kdc[6057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699317, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
I20260929 16:28:37.866091 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.842942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:50629 (local address 127.5.232.190:37111)
0929 16:28:37.843371 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:37.843376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:37.844825 (+ 1449us) server_negotiation.cc:407] Connection header received
0929 16:28:37.847704 (+ 2879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:37.847709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:37.847795 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:37.847889 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:37.850782 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.851499 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.853793 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.854625 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.854778 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.858806 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:37.858816 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:37.858824 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:37.858879 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:37.862667 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:37.863578 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:37.863585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:37.863589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:37.863761 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:37.864279 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:37.864282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:37.864284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:37.864556 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:37.864673 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:37.865511 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 16:28:37.865807 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":229,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
I20260929 16:28:37.867266 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.843607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:37111 (local address 127.5.232.130:50629)
0929 16:28:37.844595 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:37.844662 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:37.847363 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:37.848243 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:37.848257 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:37.849128 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:37.850539 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.850576 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.851705 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.851714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.853616 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.853631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.854864 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.854869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.855092 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.855868 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:37.855893 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:37.858471 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:37.862878 (+ 4407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:37.862888 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:37.862904 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:37.863391 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:37.863910 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:37.863918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:37.863922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:37.864139 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:37.864677 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:37.864684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:37.865066 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 16:28:37.865310 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 16:28:37.865829 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":365,"threads_started":1}
I20260929 16:28:37.869445 6425 heartbeater.cc:347] Connected to a master server at 127.5.232.190:37111
I20260929 16:28:37.869808 6425 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:37.870981 6425 heartbeater.cc:511] Master 127.5.232.190:37111 requested a full tablet report, sending...
I20260929 16:28:37.872627 6102 ts_manager.cc:194] Registered new tserver with Master: eba26d1729af40969f7138f09eb812b2 (127.5.232.130:45199)
I20260929 16:28:37.873979 6102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:50629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:28:37.889262 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37111
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:38.234740 6431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:38.235052 6431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:38.235255 6431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:38.243412 6431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:38.243546 6431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:38.243750 6431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:28:38.253809 6431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:38.255216 6431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:38.256630 6431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:38.270984 6437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:38.271064 6439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:38.271551 6436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:38.271898 6431 server_base.cc:1034] running on GCE node
I20260929 16:28:38.272545 6431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:38.273661 6431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:38.274917 6431 hybrid_clock.cc:648] HybridClock initialized: now 1790699318274841 us; error 67 us; skew 500 ppm
I20260929 16:28:38.275202 6431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:38 dist-test-slave-dssx krb5kdc[6057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:38.278879 6431 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:28:38.280467 6431 webserver.cc:460] Webserver started at http://127.5.232.131:39265/ using document root <none> and password file <none>
I20260929 16:28:38.281142 6431 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:38.281247 6431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:38.281615 6431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:38.284760 6431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "89e640a253a643a49176f2d7f6c7d415"
format_stamp: "Formatted at 2026-09-29 16:28:38 on dist-test-slave-dssx"
server_key: "9810fb65ed99635bdbee298e18218633"
server_key_iv: "76a5606bea77d7c388f91a075454dba1"
server_key_version: "encryptionkey@0"
I20260929 16:28:38.285620 6431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "89e640a253a643a49176f2d7f6c7d415"
format_stamp: "Formatted at 2026-09-29 16:28:38 on dist-test-slave-dssx"
server_key: "9810fb65ed99635bdbee298e18218633"
server_key_iv: "76a5606bea77d7c388f91a075454dba1"
server_key_version: "encryptionkey@0"
I20260929 16:28:38.291087 6431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 16:28:38.294962 6446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:38.296494 6431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:28:38.296761 6431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "89e640a253a643a49176f2d7f6c7d415"
format_stamp: "Formatted at 2026-09-29 16:28:38 on dist-test-slave-dssx"
server_key: "9810fb65ed99635bdbee298e18218633"
server_key_iv: "76a5606bea77d7c388f91a075454dba1"
server_key_version: "encryptionkey@0"
I20260929 16:28:38.296983 6431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:38.321549 6431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:38.325636 6431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:38.325881 6431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:38.326793 6431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:38.328078 6431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:38.328158 6431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:38.328243 6431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:38.328287 6431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:38.354306 6291 heartbeater.cc:503] Master 127.5.232.190:37111 was elected leader, sending a full tablet report...
I20260929 16:28:38.365674 6431 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:40413
I20260929 16:28:38.365731 6558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:40413 every 8 connection(s)
I20260929 16:28:38.367688 6431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:28:38.377656 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 6431
I20260929 16:28:38.377882 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:28:38.378331 6050 external_mini_cluster.cc:1428] Setting key b23ad14fc7b34971f1c403a4320bac19
Sep 29 16:28:38 dist-test-slave-dssx krb5kdc[6057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699318, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:38.390623 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:38.370629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:34671 (local address 127.5.232.190:37111)
0929 16:28:38.371108 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:38.371113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:38.372680 (+ 1567us) server_negotiation.cc:407] Connection header received
0929 16:28:38.374655 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:38.374660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:38.374781 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:38.374913 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:38.376692 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.377575 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.379557 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.380162 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.380318 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:38.384138 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:38.384145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:38.384150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:38.384195 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:38.386804 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.387641 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.387647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.387651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.387806 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.388278 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.388286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.388289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.388566 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:38.388750 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:38.389536 (+ 786us) server_negotiation.cc:299] Negotiation successful
0929 16:28:38.389780 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":235,"threads_started":1}
I20260929 16:28:38.391615 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:38.371278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:37111 (local address 127.5.232.131:34671)
0929 16:28:38.372082 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:38.372121 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:38.374341 (+ 2220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:38.375168 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:38.375177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:38.375731 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:38.376499 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:38.376527 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.377738 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.377743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:38.379400 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:38.379414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.380416 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.380425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:38.380692 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:38.381506 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:38.381529 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:38.383893 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:38.386981 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:38.386992 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:38.387014 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:38.387494 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:38.387931 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:38.387937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:38.387940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:38.388143 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:38.388704 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:38.388713 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:38.389106 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 16:28:38.389431 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 16:28:38.390001 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":330,"threads_started":1}
I20260929 16:28:38.393227 6559 heartbeater.cc:347] Connected to a master server at 127.5.232.190:37111
I20260929 16:28:38.393615 6559 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:38.394577 6559 heartbeater.cc:511] Master 127.5.232.190:37111 requested a full tablet report, sending...
I20260929 16:28:38.395998 6102 ts_manager.cc:194] Registered new tserver with Master: 89e640a253a643a49176f2d7f6c7d415 (127.5.232.131:40413)
I20260929 16:28:38.396886 6102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:34671
I20260929 16:28:38.407274 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:28:38 dist-test-slave-dssx krb5kdc[6057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:28:38 dist-test-slave-dssx krb5kdc[6057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:28:38.453550 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:38.437580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.5.232.129:34519)
0929 16:28:38.438239 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:38.438246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:38.438300 (+ 54us) server_negotiation.cc:407] Connection header received
0929 16:28:38.438360 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:38.438366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:38.438607 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:38.438729 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:38.440155 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.441183 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.442565 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.443143 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.443241 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:38.447825 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:38.447837 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:38.447853 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:38.447896 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:38.450868 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.451545 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.451552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.451555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.451679 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.452054 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.452058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.452059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.452382 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:38.452601 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:38.452852 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 16:28:38.453060 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":353,"threads_started":1}
W20260929 16:28:38.454897 6206 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46748
Sep 29 16:28:38 dist-test-slave-dssx krb5kdc[6057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:28:38 dist-test-slave-dssx krb5kdc[6057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:28:38.501164 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:38.485875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.5.232.129:34519)
0929 16:28:38.486120 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:38.486128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:38.486408 (+ 280us) server_negotiation.cc:407] Connection header received
0929 16:28:38.486634 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:38.486639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:38.486735 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:38.486852 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:38.488210 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.488916 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.490261 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.491142 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.491303 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:38.495655 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:38.495661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:38.495665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:38.495707 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:38.498452 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.499196 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.499209 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.499211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.499339 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.499743 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.499748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.499751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.500013 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:38.500290 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:38.500587 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 16:28:38.500822 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:28:38.502844 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 6163
I20260929 16:28:38.516050 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:38.503844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.5.232.129:34519)
0929 16:28:38.504011 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:38.504015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:38.504453 (+ 438us) server_negotiation.cc:407] Connection header received
0929 16:28:38.504579 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:38.504583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:38.504640 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:38.504720 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:38.505972 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.506681 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.507924 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.508314 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.508408 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:38.510812 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:38.510817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:38.510820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:38.510855 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:38.513615 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.514419 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.514423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.514425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.514561 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.514897 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.514900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.514902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.515158 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:38.515333 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:38.515612 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 16:28:38.515775 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:28:38.516968 6286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:38.601639 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 6163
I20260929 16:28:38.618366 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 6297
Sep 29 16:28:38 dist-test-slave-dssx krb5kdc[6057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:28:38.634940 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:38.619353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.5.232.130:45199)
0929 16:28:38.619870 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:38.619877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:38.619901 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:28:38.619959 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:38.619965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:38.620170 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:38.620360 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:38.621616 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.622609 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.623857 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.624402 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.624555 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:38.628825 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:38.628840 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:38.628857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:38.628896 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:38.631962 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.632687 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.632694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.632697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.632823 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.633193 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.633196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.633198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.633791 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:38.634034 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:38.634296 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 16:28:38.634522 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260929 16:28:38.636194 6420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:38.720674 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 6297
I20260929 16:28:38.742961 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 6431
Sep 29 16:28:38 dist-test-slave-dssx krb5kdc[6057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:28:38.759184 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:38.744001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.5.232.131:40413)
0929 16:28:38.744608 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:38.744615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:38.744643 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:28:38.744701 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:38.744708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:38.744926 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:38.745058 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:38.746362 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.747402 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.748695 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:38.749262 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:38.749360 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:38.753767 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:38.753781 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:38.753798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:38.753840 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:38.756457 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.757151 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.757158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.757161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.757282 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:38.757813 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:38.757817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:38.757819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:38.758169 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:38.758405 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:38.758695 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 16:28:38.758900 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":310,"threads_started":1}
I20260929 16:28:38.760526 6554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:38.845489 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 6431
I20260929 16:28:38.862149 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 6070
I20260929 16:28:38.863539 6132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:38.967686 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 6070
2026-09-29T16:28:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4063 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:28:39 dist-test-slave-dssx krb5kdc[6581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:28:39 dist-test-slave-dssx krb5kdc[6581](info): set up 2 sockets
Sep 29 16:28:39 dist-test-slave-dssx krb5kdc[6581](info): commencing operation
krb5kdc: starting...
W20260929 16:28:40.057468 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:28:40 dist-test-slave-dssx krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:28:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:28:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:28:40.245585 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:36781
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:40.574852 6594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:40.575266 6594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:40.575441 6594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:40.583194 6594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:40.583370 6594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:40.583493 6594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:40.583600 6594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:40.583679 6594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:40.594065 6594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:36781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:40.595504 6594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:40.596908 6594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:40.609894 6599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:40.609926 6602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:40.609973 6600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:40.610669 6594 server_base.cc:1034] running on GCE node
I20260929 16:28:40.611625 6594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:40.613147 6594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:40.614400 6594 hybrid_clock.cc:648] HybridClock initialized: now 1790699320614361 us; error 57 us; skew 500 ppm
I20260929 16:28:40.614787 6594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:40 dist-test-slave-dssx krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:40.618005 6594 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:28:40.619566 6594 webserver.cc:460] Webserver started at http://127.5.232.190:40175/ using document root <none> and password file <none>
I20260929 16:28:40.620318 6594 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:40.620471 6594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:40.620790 6594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:40.623965 6594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "4fd56e674aa042ea801a52076fe0697a"
format_stamp: "Formatted at 2026-09-29 16:28:40 on dist-test-slave-dssx"
server_key: "452eea7bc13dfcbab6986f95264805b5"
server_key_iv: "728f57e63875ba8779fd1e71e9833bd6"
server_key_version: "encryptionkey@0"
I20260929 16:28:40.624766 6594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "4fd56e674aa042ea801a52076fe0697a"
format_stamp: "Formatted at 2026-09-29 16:28:40 on dist-test-slave-dssx"
server_key: "452eea7bc13dfcbab6986f95264805b5"
server_key_iv: "728f57e63875ba8779fd1e71e9833bd6"
server_key_version: "encryptionkey@0"
I20260929 16:28:40.631932 6594 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.001s
I20260929 16:28:40.636899 6609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:40.638379 6594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:40.638602 6594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "4fd56e674aa042ea801a52076fe0697a"
format_stamp: "Formatted at 2026-09-29 16:28:40 on dist-test-slave-dssx"
server_key: "452eea7bc13dfcbab6986f95264805b5"
server_key_iv: "728f57e63875ba8779fd1e71e9833bd6"
server_key_version: "encryptionkey@0"
I20260929 16:28:40.638819 6594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:40.661747 6594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:40.665880 6594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:40.666309 6594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:40.685607 6594 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:36781
I20260929 16:28:40.685642 6660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:36781 every 8 connection(s)
I20260929 16:28:40.687111 6594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:40.690999 6661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:40.694038 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 6594
I20260929 16:28:40.694280 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:28:40.695086 6050 external_mini_cluster.cc:1428] Setting key 6f04c051eb17d6909cb245bf0c622f9f
I20260929 16:28:40.704754 6661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a: Bootstrap starting.
I20260929 16:28:40.709043 6661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:40.710568 6661 log.cc:824] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:28:40 dist-test-slave-dssx krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:40.713982 6661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a: No bootstrap required, opened a new log
I20260929 16:28:40.720296 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.697474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.5.232.190:36781)
0929 16:28:40.698316 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.698329 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.698376 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:28:40.700263 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.700290 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.700769 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.701156 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.703116 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.704491 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.706377 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.707155 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.707307 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.712462 (+ 5155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.712471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.712487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.712529 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.715679 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.716469 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.716477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.716484 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.716667 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.717084 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.717088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.717090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.717730 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.718059 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.718601 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.719047 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":407,"threads_started":1}
I20260929 16:28:40.722378 6661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd56e674aa042ea801a52076fe0697a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36781 } }
I20260929 16:28:40.722779 6661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:40.722911 6661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd56e674aa042ea801a52076fe0697a, State: Initialized, Role: FOLLOWER
I20260929 16:28:40.723577 6661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd56e674aa042ea801a52076fe0697a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36781 } }
I20260929 16:28:40.723827 6661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:40.723951 6661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:40.724143 6661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:40.726468 6661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd56e674aa042ea801a52076fe0697a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36781 } }
I20260929 16:28:40.727047 6661 leader_election.cc:302] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fd56e674aa042ea801a52076fe0697a; no voters:
I20260929 16:28:40.727789 6661 leader_election.cc:288] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:28:40.727903 6666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:40.728569 6666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [term 1 LEADER]: Becoming Leader. State: Replica: 4fd56e674aa042ea801a52076fe0697a, State: Running, Role: LEADER
I20260929 16:28:40.729161 6666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd56e674aa042ea801a52076fe0697a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36781 } }
I20260929 16:28:40.730302 6661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:40.732273 6668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fd56e674aa042ea801a52076fe0697a. Latest consensus state: current_term: 1 leader_uuid: "4fd56e674aa042ea801a52076fe0697a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd56e674aa042ea801a52076fe0697a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36781 } } }
I20260929 16:28:40.732683 6668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:40.733479 6667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fd56e674aa042ea801a52076fe0697a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd56e674aa042ea801a52076fe0697a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36781 } } }
I20260929 16:28:40.734870 6667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:40.735054 6674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:40.740559 6674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:40.750489 6674 catalog_manager.cc:1347] Generated new cluster ID: a37dd8e8c32d451d86799b8edba599d0
I20260929 16:28:40.750630 6674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:40.768366 6674 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:28:40.769954 6674 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:40.777591 6674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fd56e674aa042ea801a52076fe0697a: Generated new TSK 0
I20260929 16:28:40.778762 6674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:28:40.833351 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:36781
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:41.183652 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:41.183976 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:41.184087 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:41.191751 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:41.191891 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:41.192020 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:28:41.202502 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:36781
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:41.203871 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:41.205252 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:41.221827 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:41.221747 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:41.221835 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:41.223220 6687 server_base.cc:1034] running on GCE node
I20260929 16:28:41.223932 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:41.225065 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:41.226293 6687 hybrid_clock.cc:648] HybridClock initialized: now 1790699321226210 us; error 86 us; skew 500 ppm
I20260929 16:28:41.226673 6687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:41 dist-test-slave-dssx krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699321, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:41.230285 6687 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:28:41.232069 6687 webserver.cc:460] Webserver started at http://127.5.232.129:33569/ using document root <none> and password file <none>
I20260929 16:28:41.232864 6687 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:41.233042 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:41.233446 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:41.236717 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "001b4912a5724e7d9f772756d272a4c0"
format_stamp: "Formatted at 2026-09-29 16:28:41 on dist-test-slave-dssx"
server_key: "bb3d08064101ab5a9d21d0e9330d5d6e"
server_key_iv: "1e1718602cd40aafd978a4b839c6e53c"
server_key_version: "encryptionkey@0"
I20260929 16:28:41.237725 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "001b4912a5724e7d9f772756d272a4c0"
format_stamp: "Formatted at 2026-09-29 16:28:41 on dist-test-slave-dssx"
server_key: "bb3d08064101ab5a9d21d0e9330d5d6e"
server_key_iv: "1e1718602cd40aafd978a4b839c6e53c"
server_key_version: "encryptionkey@0"
I20260929 16:28:41.243700 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:28:41.247838 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:41.249518 6687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:41.249728 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "001b4912a5724e7d9f772756d272a4c0"
format_stamp: "Formatted at 2026-09-29 16:28:41 on dist-test-slave-dssx"
server_key: "bb3d08064101ab5a9d21d0e9330d5d6e"
server_key_iv: "1e1718602cd40aafd978a4b839c6e53c"
server_key_version: "encryptionkey@0"
I20260929 16:28:41.249979 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:41.270284 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:41.275355 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:41.275707 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:41.277009 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:41.278680 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:41.278764 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:41.278934 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:41.278999 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:41.314160 6687 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:34211
I20260929 16:28:41.314203 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:34211 every 8 connection(s)
I20260929 16:28:41.315580 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:28:41.319522 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 6687
I20260929 16:28:41.319754 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:28:41.320515 6050 external_mini_cluster.cc:1428] Setting key 9117222c6b2b8170b70bfac319277744
Sep 29 16:28:41 dist-test-slave-dssx krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699321, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
I20260929 16:28:41.338387 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:41.318441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:60499 (local address 127.5.232.190:36781)
0929 16:28:41.318919 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:41.318942 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:41.320119 (+ 1177us) server_negotiation.cc:407] Connection header received
0929 16:28:41.322303 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:41.322308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:41.322395 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:41.322539 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:41.324219 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:41.324977 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:41.326778 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:41.327190 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:41.327353 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:41.331639 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:41.331645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:41.331649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:41.331687 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:41.334503 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:41.335731 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:41.335736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:41.335739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:41.335872 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:41.336457 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:41.336460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:41.336462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:41.336723 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:41.336833 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:41.337936 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0929 16:28:41.338130 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":275,"threads_started":1}
I20260929 16:28:41.339512 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:41.319006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36781 (local address 127.5.232.129:60499)
0929 16:28:41.319931 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:41.319988 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:41.321998 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:41.322812 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:41.322821 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:41.323375 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:41.324002 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:41.324021 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:41.325170 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:41.325181 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:41.326588 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:41.326603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:41.327455 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:41.327464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:41.327721 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:41.328503 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:41.328532 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:41.331371 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:41.334729 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:41.334744 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:41.334770 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:41.335575 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:41.336020 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:41.336026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:41.336030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:41.336317 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:41.336876 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:41.336887 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:41.337463 (+ 576us) client_negotiation.cc:769] Sending connection context
0929 16:28:41.337839 (+ 376us) client_negotiation.cc:240] Negotiation successful
0929 16:28:41.338326 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":338,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
I20260929 16:28:41.341540 6815 heartbeater.cc:347] Connected to a master server at 127.5.232.190:36781
I20260929 16:28:41.341917 6815 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:41.342818 6815 heartbeater.cc:511] Master 127.5.232.190:36781 requested a full tablet report, sending...
I20260929 16:28:41.345175 6626 ts_manager.cc:194] Registered new tserver with Master: 001b4912a5724e7d9f772756d272a4c0 (127.5.232.129:34211)
I20260929 16:28:41.346747 6626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:60499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:28:41.364590 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:36781
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:41.702594 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:41.702891 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:41.703061 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:41.710752 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:41.710891 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:41.711098 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:28:41.722394 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:36781
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:41.724341 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:41.726202 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:41.741640 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:41.741592 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:41.741694 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:41.742349 6821 server_base.cc:1034] running on GCE node
I20260929 16:28:41.743081 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:41.744232 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:41.745494 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790699321745438 us; error 76 us; skew 500 ppm
I20260929 16:28:41.745841 6821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:41 dist-test-slave-dssx krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699321, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:41.749122 6821 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:28:41.750831 6821 webserver.cc:460] Webserver started at http://127.5.232.130:45053/ using document root <none> and password file <none>
I20260929 16:28:41.751588 6821 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:41.751753 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:41.752074 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:41.755218 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "da0602c66fca4ee28b9579705f0696f8"
format_stamp: "Formatted at 2026-09-29 16:28:41 on dist-test-slave-dssx"
server_key: "8f6db8b851576a175e8f722e70f5cdaa"
server_key_iv: "a24400c44f5e76770e62caaf9eaff1ad"
server_key_version: "encryptionkey@0"
I20260929 16:28:41.756047 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "da0602c66fca4ee28b9579705f0696f8"
format_stamp: "Formatted at 2026-09-29 16:28:41 on dist-test-slave-dssx"
server_key: "8f6db8b851576a175e8f722e70f5cdaa"
server_key_iv: "a24400c44f5e76770e62caaf9eaff1ad"
server_key_version: "encryptionkey@0"
I20260929 16:28:41.761620 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 16:28:41.765342 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:41.766901 6821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:41.767113 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "da0602c66fca4ee28b9579705f0696f8"
format_stamp: "Formatted at 2026-09-29 16:28:41 on dist-test-slave-dssx"
server_key: "8f6db8b851576a175e8f722e70f5cdaa"
server_key_iv: "a24400c44f5e76770e62caaf9eaff1ad"
server_key_version: "encryptionkey@0"
I20260929 16:28:41.767343 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:41.789621 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:41.794461 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:41.794790 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:41.795809 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:41.797194 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:41.797313 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:41.797487 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:41.797564 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:41.833693 6821 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:34357
I20260929 16:28:41.833727 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:34357 every 8 connection(s)
I20260929 16:28:41.835338 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:28:41.844162 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 6821
I20260929 16:28:41.844388 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:28:41.844955 6050 external_mini_cluster.cc:1428] Setting key a54792927b7d403d74a558045adfe780
Sep 29 16:28:41 dist-test-slave-dssx krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699321, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:41.864849 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:41.838903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:57017 (local address 127.5.232.190:36781)
0929 16:28:41.840164 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:41.840173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:41.840795 (+ 622us) server_negotiation.cc:407] Connection header received
0929 16:28:41.844688 (+ 3893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:41.844698 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:41.844803 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:41.844928 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:41.847270 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:41.848408 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:41.851297 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:41.852078 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:41.852257 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:41.858218 (+ 5961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:41.858224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:41.858228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:41.858267 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:41.861491 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:41.862406 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:41.862413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:41.862416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:41.862564 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:41.863041 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:41.863046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:41.863049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:41.863336 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:41.863496 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:41.864272 (+ 776us) server_negotiation.cc:299] Negotiation successful
0929 16:28:41.864503 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095}
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
I20260929 16:28:41.866109 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:41.839584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36781 (local address 127.5.232.130:57017)
0929 16:28:41.840588 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:41.840658 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:41.844284 (+ 3626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:41.845283 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:41.845299 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:41.846192 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:41.847050 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:41.847077 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:41.848617 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:41.848626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:41.851068 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:41.851091 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:41.852633 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:41.852646 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:41.853083 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:41.854083 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:41.854126 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:41.857860 (+ 3734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:41.861701 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:41.861712 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:41.861733 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:41.862244 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:41.862691 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:41.862697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:41.862700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:41.862907 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:41.863474 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:41.863483 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:41.863877 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 16:28:41.864167 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 16:28:41.864755 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":443,"threads_started":1}
I20260929 16:28:41.867859 6949 heartbeater.cc:347] Connected to a master server at 127.5.232.190:36781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
I20260929 16:28:41.868312 6949 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:41.869416 6949 heartbeater.cc:511] Master 127.5.232.190:36781 requested a full tablet report, sending...
I20260929 16:28:41.871490 6626 ts_manager.cc:194] Registered new tserver with Master: da0602c66fca4ee28b9579705f0696f8 (127.5.232.130:34357)
I20260929 16:28:41.872839 6626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:57017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:28:41.893877 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:36781
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:42.231345 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:42.231716 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:42.231894 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:42.239737 6954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:42.239928 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:42.240129 6954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:28:42.250502 6954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:36781
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:42.251937 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:42.253490 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:42.267879 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:42.267872 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:42.267903 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:42.268777 6954 server_base.cc:1034] running on GCE node
I20260929 16:28:42.269536 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:42.270634 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:42.271852 6954 hybrid_clock.cc:648] HybridClock initialized: now 1790699322271774 us; error 74 us; skew 500 ppm
I20260929 16:28:42.272281 6954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:42 dist-test-slave-dssx krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699322, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:42.279255 6954 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:28:42.280989 6954 webserver.cc:460] Webserver started at http://127.5.232.131:40855/ using document root <none> and password file <none>
I20260929 16:28:42.281816 6954 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:42.281953 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:42.282230 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:42.285228 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "19bb0f600dc9499f83f5a2ecdc75b769"
format_stamp: "Formatted at 2026-09-29 16:28:42 on dist-test-slave-dssx"
server_key: "69112fde92320ed7455bb987bb9a622c"
server_key_iv: "76d5108fb004bf4614629fcba897fe92"
server_key_version: "encryptionkey@0"
I20260929 16:28:42.286145 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "19bb0f600dc9499f83f5a2ecdc75b769"
format_stamp: "Formatted at 2026-09-29 16:28:42 on dist-test-slave-dssx"
server_key: "69112fde92320ed7455bb987bb9a622c"
server_key_iv: "76d5108fb004bf4614629fcba897fe92"
server_key_version: "encryptionkey@0"
I20260929 16:28:42.291754 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:28:42.295660 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:42.297278 6954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:28:42.297542 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "19bb0f600dc9499f83f5a2ecdc75b769"
format_stamp: "Formatted at 2026-09-29 16:28:42 on dist-test-slave-dssx"
server_key: "69112fde92320ed7455bb987bb9a622c"
server_key_iv: "76d5108fb004bf4614629fcba897fe92"
server_key_version: "encryptionkey@0"
I20260929 16:28:42.297782 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:42.315580 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:42.320068 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:42.320410 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:42.321514 6954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:42.322954 6954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:42.323073 6954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:42.323236 6954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:42.323310 6954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:42.350070 6815 heartbeater.cc:503] Master 127.5.232.190:36781 was elected leader, sending a full tablet report...
I20260929 16:28:42.359073 6954 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:46779
I20260929 16:28:42.359127 7081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:46779 every 8 connection(s)
I20260929 16:28:42.360503 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:28:42.362628 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 6954
I20260929 16:28:42.362841 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:28:42.363293 6050 external_mini_cluster.cc:1428] Setting key 433b05f4b81824fd6f7193ad91b04806
Sep 29 16:28:42 dist-test-slave-dssx krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699322, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:42.383123 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.363767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:47919 (local address 127.5.232.190:36781)
0929 16:28:42.363987 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.363994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.365475 (+ 1481us) server_negotiation.cc:407] Connection header received
0929 16:28:42.367679 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.367683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.367782 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.367870 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:42.369561 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.370398 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.372189 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.372676 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.372781 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.376558 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:42.376564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:42.376568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:42.376606 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:42.379927 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.380839 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.380843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.380845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.380976 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.381531 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.381535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.381538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.381810 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:42.381944 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.382690 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.382866 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:28:42.384299 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.364396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36781 (local address 127.5.232.131:47919)
0929 16:28:42.365275 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:42.365324 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:42.367367 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:42.368119 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:42.368128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:42.368689 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:42.369284 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.369303 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.370560 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.370564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.372036 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.372049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.372893 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.372899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.373126 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.373877 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:42.373904 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:42.376288 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:42.380206 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:42.380215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:42.380231 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:42.380714 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:42.381103 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:42.381107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:42.381109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:42.381321 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:42.381946 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:42.381955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:42.382319 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 16:28:42.382568 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 16:28:42.382989 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":388,"threads_started":1}
I20260929 16:28:42.385826 7082 heartbeater.cc:347] Connected to a master server at 127.5.232.190:36781
I20260929 16:28:42.386174 7082 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:42.386940 7082 heartbeater.cc:511] Master 127.5.232.190:36781 requested a full tablet report, sending...
I20260929 16:28:42.388406 6626 ts_manager.cc:194] Registered new tserver with Master: 19bb0f600dc9499f83f5a2ecdc75b769 (127.5.232.131:46779)
I20260929 16:28:42.389253 6626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:47919
I20260929 16:28:42.391829 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:28:42.412385 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.399009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.5.232.190:36781)
0929 16:28:42.399205 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.399210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.399719 (+ 509us) server_negotiation.cc:407] Connection header received
0929 16:28:42.399858 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.399862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.399948 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.400050 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:42.401758 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.402666 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.404131 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.404526 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.404637 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.407154 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:42.407162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:42.407168 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:42.407222 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:42.409941 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.410665 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.410670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.410672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.410803 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.411151 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.411154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.411156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.411474 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:42.411655 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.411968 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.412153 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:28:42.419364 6626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:28:42.426913 6626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:42.464658 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.439345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:34357 (local address 127.0.0.1:52460)
0929 16:28:42.454116 (+ 14771us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:42.454166 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:42.454365 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:42.455756 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:42.455768 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:42.455811 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:42.456324 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.456343 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.458697 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.458702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.460879 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.460890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.463342 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.463353 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.463595 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.463743 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 16:28:42.463896 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 16:28:42.464126 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14516,"mutex_wait_us":64,"thread_start_us":18088,"threads_started":1}
I20260929 16:28:42.465557 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.439335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.5.232.130:34357)
0929 16:28:42.452299 (+ 12964us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.452312 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.454635 (+ 2323us) server_negotiation.cc:407] Connection header received
0929 16:28:42.454775 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.454788 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.455154 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.455383 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:42.456627 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.458534 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.461068 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.462482 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.462627 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.462854 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.465072 (+ 2218us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.465221 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12721,"thread_start_us":12976,"threads_started":1}
I20260929 16:28:42.467384 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.442105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:46779 (local address 127.0.0.1:56710)
0929 16:28:42.457736 (+ 15631us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:42.457755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:42.457876 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:42.459096 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:42.459103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:42.459132 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:42.459754 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.459767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.461860 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.461866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.464529 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.464543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.466545 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.466551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.466726 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.466766 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:28:42.466861 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:28:42.467005 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15487,"mutex_wait_us":54,"thread_start_us":11610,"threads_started":1}
I20260929 16:28:42.467443 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.442727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.5.232.131:46779)
0929 16:28:42.453058 (+ 10331us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.453069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.458401 (+ 5332us) server_negotiation.cc:407] Connection header received
0929 16:28:42.458527 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.458537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.458834 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.459073 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:42.459982 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.461709 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.464723 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.466322 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.466493 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.466733 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.466940 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.467123 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10140,"thread_start_us":9989,"threads_started":1}
I20260929 16:28:42.468358 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.439474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34211 (local address 127.0.0.1:36638)
0929 16:28:42.456777 (+ 17303us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:42.456803 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:42.456955 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:42.459174 (+ 2219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:42.459181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:42.459209 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:42.459951 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.459964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.462566 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.462571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.465295 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.465309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.467561 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.467571 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.467749 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.467787 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:28:42.467880 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 16:28:42.468021 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17130,"mutex_wait_us":109,"thread_start_us":17591,"threads_started":1}
I20260929 16:28:42.469081 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.439609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.5.232.129:34211)
0929 16:28:42.458499 (+ 18890us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.458509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.458552 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:28:42.458656 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.458664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.458934 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.459356 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:42.460157 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.462392 (+ 2235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.465503 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.467056 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.467198 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.467430 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.468590 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.468769 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18664,"thread_start_us":18496,"threads_started":1}
I20260929 16:28:42.469892 6884 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4713b1b0fe4de09a1eff309fa1cc85 (DEFAULT_TABLE table=test-table [id=277f4b94c57a4b3791e3f95e8541bb1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:42.471506 6884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4713b1b0fe4de09a1eff309fa1cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:42.471397 7017 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4713b1b0fe4de09a1eff309fa1cc85 (DEFAULT_TABLE table=test-table [id=277f4b94c57a4b3791e3f95e8541bb1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:42.472806 7017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4713b1b0fe4de09a1eff309fa1cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:42.472640 6750 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4713b1b0fe4de09a1eff309fa1cc85 (DEFAULT_TABLE table=test-table [id=277f4b94c57a4b3791e3f95e8541bb1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:42.474069 6750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4713b1b0fe4de09a1eff309fa1cc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:42.485147 7100 tablet_bootstrap.cc:492] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769: Bootstrap starting.
I20260929 16:28:42.489662 7100 tablet_bootstrap.cc:654] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:42.490145 7102 tablet_bootstrap.cc:492] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0: Bootstrap starting.
I20260929 16:28:42.491293 7100 log.cc:824] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:42.492236 7101 tablet_bootstrap.cc:492] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8: Bootstrap starting.
I20260929 16:28:42.494904 7102 tablet_bootstrap.cc:654] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:42.496150 7101 tablet_bootstrap.cc:654] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:42.496253 7102 log.cc:824] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:42.496709 7100 tablet_bootstrap.cc:492] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769: No bootstrap required, opened a new log
I20260929 16:28:42.497051 7100 ts_tablet_manager.cc:1397] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260929 16:28:42.497452 7101 log.cc:824] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:42.500267 7102 tablet_bootstrap.cc:492] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0: No bootstrap required, opened a new log
I20260929 16:28:42.500572 7102 ts_tablet_manager.cc:1397] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260929 16:28:42.500665 7101 tablet_bootstrap.cc:492] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8: No bootstrap required, opened a new log
I20260929 16:28:42.500900 7101 ts_tablet_manager.cc:1397] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:28:42.508769 7101 raft_consensus.cc:348] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } }
I20260929 16:28:42.508809 7100 raft_consensus.cc:348] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } }
I20260929 16:28:42.509231 7101 raft_consensus.cc:374] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:42.509281 7100 raft_consensus.cc:374] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:42.509325 7101 raft_consensus.cc:729] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0602c66fca4ee28b9579705f0696f8, State: Initialized, Role: FOLLOWER
I20260929 16:28:42.509366 7100 raft_consensus.cc:729] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bb0f600dc9499f83f5a2ecdc75b769, State: Initialized, Role: FOLLOWER
I20260929 16:28:42.510020 7101 consensus_queue.cc:260] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } }
I20260929 16:28:42.510136 7100 consensus_queue.cc:260] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } }
I20260929 16:28:42.512835 7082 heartbeater.cc:503] Master 127.5.232.190:36781 was elected leader, sending a full tablet report...
I20260929 16:28:42.513463 6949 heartbeater.cc:503] Master 127.5.232.190:36781 was elected leader, sending a full tablet report...
I20260929 16:28:42.513171 7102 raft_consensus.cc:348] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } }
I20260929 16:28:42.513650 7102 raft_consensus.cc:374] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:42.513743 7102 raft_consensus.cc:729] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b4912a5724e7d9f772756d272a4c0, State: Initialized, Role: FOLLOWER
I20260929 16:28:42.514160 7101 ts_tablet_manager.cc:1428] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8: Time spent starting tablet: real 0.013s user 0.006s sys 0.008s
I20260929 16:28:42.514462 7102 consensus_queue.cc:260] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } }
I20260929 16:28:42.515457 7100 ts_tablet_manager.cc:1428] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769: Time spent starting tablet: real 0.018s user 0.009s sys 0.009s
I20260929 16:28:42.517082 7102 ts_tablet_manager.cc:1428] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260929 16:28:42.568267 6816 tablet.cc:2369] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:28:42.587773 6950 tablet.cc:2369] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:28:42.612483 7083 tablet.cc:2369] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:28:42.709249 7108 raft_consensus.cc:482] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:42.709641 7108 raft_consensus.cc:504] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } }
I20260929 16:28:42.711457 7108 leader_election.cc:288] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da0602c66fca4ee28b9579705f0696f8 (127.5.232.130:34357), 19bb0f600dc9499f83f5a2ecdc75b769 (127.5.232.131:46779)
I20260929 16:28:42.721592 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.712029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:34097 (local address 127.5.232.131:46779)
0929 16:28:42.712397 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.712403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.712824 (+ 421us) server_negotiation.cc:407] Connection header received
0929 16:28:42.713032 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.713039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.713120 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.713228 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:42.714091 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.715203 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.718476 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.720162 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.720380 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.720459 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.721142 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.721286 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260929 16:28:42.721592 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.711967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:59333 (local address 127.5.232.130:34357)
0929 16:28:42.712226 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.712235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.712840 (+ 605us) server_negotiation.cc:407] Connection header received
0929 16:28:42.713032 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.713039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.713137 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.713228 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:42.714094 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.715203 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.718466 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.720233 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.720380 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.720459 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.721142 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.721286 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:28:42.721794 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.711819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:46779 (local address 127.5.232.129:34097)
0929 16:28:42.712652 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:42.712673 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:42.712851 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:42.713278 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:42.713283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:42.713342 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:42.713950 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.713963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.715393 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.715397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.718215 (+ 2818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.718234 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.720592 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.720601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.720876 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.720959 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 16:28:42.721074 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:28:42.721248 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":443,"spinlock_wait_cycles":8960,"thread_start_us":400,"threads_started":1}
I20260929 16:28:42.722193 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.711957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:34357 (local address 127.5.232.129:59333)
0929 16:28:42.712714 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:42.712735 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:42.712889 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:42.713278 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:42.713283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:42.713340 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:42.713868 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.713884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.715391 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.715397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.718215 (+ 2818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:42.718234 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.720613 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.720619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:42.720878 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.720972 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 16:28:42.721074 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:28:42.721248 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":277,"spinlock_wait_cycles":71680,"thread_start_us":455,"threads_started":1}
I20260929 16:28:42.723104 7037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4713b1b0fe4de09a1eff309fa1cc85" candidate_uuid: "001b4912a5724e7d9f772756d272a4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" is_pre_election: true
I20260929 16:28:42.723109 6904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4713b1b0fe4de09a1eff309fa1cc85" candidate_uuid: "001b4912a5724e7d9f772756d272a4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0602c66fca4ee28b9579705f0696f8" is_pre_election: true
I20260929 16:28:42.723691 7037 raft_consensus.cc:2454] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b4912a5724e7d9f772756d272a4c0 in term 0.
I20260929 16:28:42.723691 6904 raft_consensus.cc:2454] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b4912a5724e7d9f772756d272a4c0 in term 0.
I20260929 16:28:42.724542 6706 leader_election.cc:302] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 001b4912a5724e7d9f772756d272a4c0, 19bb0f600dc9499f83f5a2ecdc75b769, da0602c66fca4ee28b9579705f0696f8; no voters:
I20260929 16:28:42.725039 7108 raft_consensus.cc:2781] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:42.725164 7108 raft_consensus.cc:482] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:42.725302 7108 raft_consensus.cc:3037] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:42.727526 7108 raft_consensus.cc:504] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } }
I20260929 16:28:42.728233 7108 leader_election.cc:288] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [CANDIDATE]: Term 1 election: Requested vote from peers da0602c66fca4ee28b9579705f0696f8 (127.5.232.130:34357), 19bb0f600dc9499f83f5a2ecdc75b769 (127.5.232.131:46779)
I20260929 16:28:42.728742 6904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4713b1b0fe4de09a1eff309fa1cc85" candidate_uuid: "001b4912a5724e7d9f772756d272a4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0602c66fca4ee28b9579705f0696f8"
I20260929 16:28:42.728787 7037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4713b1b0fe4de09a1eff309fa1cc85" candidate_uuid: "001b4912a5724e7d9f772756d272a4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb0f600dc9499f83f5a2ecdc75b769"
I20260929 16:28:42.728976 6904 raft_consensus.cc:3037] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:42.728999 7037 raft_consensus.cc:3037] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:42.731424 7037 raft_consensus.cc:2454] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 19bb0f600dc9499f83f5a2ecdc75b769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b4912a5724e7d9f772756d272a4c0 in term 1.
I20260929 16:28:42.731439 6904 raft_consensus.cc:2454] T 0f4713b1b0fe4de09a1eff309fa1cc85 P da0602c66fca4ee28b9579705f0696f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b4912a5724e7d9f772756d272a4c0 in term 1.
I20260929 16:28:42.732010 6704 leader_election.cc:302] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 001b4912a5724e7d9f772756d272a4c0, 19bb0f600dc9499f83f5a2ecdc75b769, da0602c66fca4ee28b9579705f0696f8; no voters:
I20260929 16:28:42.732405 7108 raft_consensus.cc:2781] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:42.732908 7108 raft_consensus.cc:686] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 001b4912a5724e7d9f772756d272a4c0, State: Running, Role: LEADER
I20260929 16:28:42.733498 7108 consensus_queue.cc:237] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } }
I20260929 16:28:42.739130 6624 catalog_manager.cc:5495] T 0f4713b1b0fe4de09a1eff309fa1cc85 P 001b4912a5724e7d9f772756d272a4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001b4912a5724e7d9f772756d272a4c0 (127.5.232.129). New cstate: current_term: 1 leader_uuid: "001b4912a5724e7d9f772756d272a4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da0602c66fca4ee28b9579705f0696f8" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 34357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b4912a5724e7d9f772756d272a4c0" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19bb0f600dc9499f83f5a2ecdc75b769" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46779 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:28:42 dist-test-slave-dssx krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:28:42.833024 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.816637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.5.232.129:34211)
0929 16:28:42.816969 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.816976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.817288 (+ 312us) server_negotiation.cc:407] Connection header received
0929 16:28:42.817418 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.817422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.817490 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.817580 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:42.819076 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.819966 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.821236 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.821746 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.821858 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.826570 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:42.826586 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:42.826603 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:42.826649 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:42.829908 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.830823 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.830832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.830834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.830982 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.831338 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.831341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.831343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.831820 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:42.832060 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.832420 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.832678 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
Sep 29 16:28:42 dist-test-slave-dssx krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699322, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:28:42 dist-test-slave-dssx krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699322, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:28:42.882292 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.868328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.5.232.129:34211)
0929 16:28:42.868588 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.868596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.868916 (+ 320us) server_negotiation.cc:407] Connection header received
0929 16:28:42.869162 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.869166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.869230 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.869351 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:42.870788 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.871605 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.872888 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.873319 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.873531 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.877501 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:42.877507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:42.877510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:42.877552 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:42.879897 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.880564 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.880568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.880570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.880688 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.881040 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.881043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.881045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.881306 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:42.881604 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.881817 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.882017 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 16:28:42 dist-test-slave-dssx krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:28:42 dist-test-slave-dssx krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:28:42.930739 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.915128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.5.232.129:34211)
0929 16:28:42.915359 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.915367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.915581 (+ 214us) server_negotiation.cc:407] Connection header received
0929 16:28:42.915896 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.915900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.915967 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.916130 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:42.917502 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.918269 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.919571 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.920000 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.920157 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.924536 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:42.924542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:42.924545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:42.924585 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:42.927732 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.928557 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.928563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.928567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.928713 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.929131 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.929137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.929140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.929537 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:42.929812 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.930137 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.930357 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260929 16:28:42.932801 6730 tablet_service.cc:2908] Rejecting scan request for tablet 0f4713b1b0fe4de09a1eff309fa1cc85: Uninitialized: clean time has not yet been initialized
I20260929 16:28:42.936007 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 6687
I20260929 16:28:42.949676 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:42.936964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.5.232.129:34211)
0929 16:28:42.937216 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:42.937226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:42.937604 (+ 378us) server_negotiation.cc:407] Connection header received
0929 16:28:42.937724 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:42.937728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:42.937793 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:42.937883 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:42.939044 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.939770 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.940976 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:42.941464 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:42.941580 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:42.944203 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:42.944208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:42.944211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:42.944248 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:42.947049 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.947795 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.947800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.947802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.947931 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:42.948267 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:42.948269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:42.948271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:42.948512 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:42.948700 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:42.949009 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 16:28:42.949309 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:28:42.950431 6810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:43.044394 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 6687
I20260929 16:28:43.062105 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 6821
Sep 29 16:28:43 dist-test-slave-dssx krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:28:43.076960 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:43.062923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.5.232.130:34357)
0929 16:28:43.063097 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:43.063101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:43.063136 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:28:43.063311 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:43.063316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:43.063388 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:43.063473 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:43.064595 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:43.065327 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:43.066607 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:43.067011 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:43.067109 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:43.071387 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:43.071404 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:43.071420 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:43.071460 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:43.074321 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:43.075055 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:43.075064 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:43.075067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:43.075220 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:43.075558 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:43.075561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:43.075562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:43.075927 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:43.076168 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:43.076407 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 16:28:43.076574 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:28:43.077759 6944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:43.169660 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 6821
I20260929 16:28:43.186383 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 6954
Sep 29 16:28:43 dist-test-slave-dssx krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:28:43.201304 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:43.187190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.5.232.131:46779)
0929 16:28:43.187364 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:43.187369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:43.187388 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:28:43.187445 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:43.187448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:43.187514 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:43.187604 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:43.188834 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:43.189663 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:43.191003 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:43.191380 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:43.191490 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:43.195918 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:43.195934 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:43.195951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:43.195992 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:43.198722 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:43.199456 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:43.199464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:43.199467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:43.199606 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:43.199946 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:43.199949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:43.199951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:43.200315 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:43.200536 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:43.200834 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 16:28:43.201029 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:28:43.202368 7077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:43.292389 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 6954
I20260929 16:28:43.309118 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 6594
I20260929 16:28:43.310451 6656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:43.418732 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 6594
2026-09-29T16:28:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4452 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:28:43 dist-test-slave-dssx krb5kdc[7130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:28:43 dist-test-slave-dssx krb5kdc[7130](info): set up 2 sockets
Sep 29 16:28:43 dist-test-slave-dssx krb5kdc[7130](info): commencing operation
krb5kdc: starting...
W20260929 16:28:44.505460 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:28:44 dist-test-slave-dssx krb5kdc[7130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:28:44.696709 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40169
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:40169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:45.025410 7143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:45.025727 7143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:45.025902 7143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:45.033224 7143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:45.033344 7143 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:28:45.033461 7143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:45.033586 7143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:45.033687 7143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:45.033803 7143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:45.044064 7143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:40169
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:45.045512 7143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:45.046969 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:45.059132 7148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.059104 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.059134 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:45.060002 7143 server_base.cc:1034] running on GCE node
I20260929 16:28:45.060832 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:45.062443 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:45.063700 7143 hybrid_clock.cc:648] HybridClock initialized: now 1790699325063601 us; error 59 us; skew 500 ppm
I20260929 16:28:45.064018 7143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:45 dist-test-slave-dssx krb5kdc[7130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:45.067265 7143 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:28:45.068704 7143 webserver.cc:460] Webserver started at http://127.5.232.190:42169/ using document root <none> and password file <none>
I20260929 16:28:45.069432 7143 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:45.069537 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:45.069821 7143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:45.072800 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "ea760c2817384d779a55a3250a0fe800"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-dssx"
server_key: "b00578aa51fd16d555ebbe00572d5ced"
server_key_iv: "57bc46de6065fea431c61fba509bbf51"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.073791 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "ea760c2817384d779a55a3250a0fe800"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-dssx"
server_key: "b00578aa51fd16d555ebbe00572d5ced"
server_key_iv: "57bc46de6065fea431c61fba509bbf51"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.079385 7143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:28:45.083238 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.084854 7143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:28:45.085067 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "ea760c2817384d779a55a3250a0fe800"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-dssx"
server_key: "b00578aa51fd16d555ebbe00572d5ced"
server_key_iv: "57bc46de6065fea431c61fba509bbf51"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.085336 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:45.098026 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:45.102854 7143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:45.103174 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:45.126428 7143 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:40169
I20260929 16:28:45.126435 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:40169 every 8 connection(s)
I20260929 16:28:45.127921 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:45.131909 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:45.135366 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 7143
I20260929 16:28:45.135599 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:28:45.136242 6050 external_mini_cluster.cc:1428] Setting key 9a2f52807bd73cff7fc1942a7d0776c7
I20260929 16:28:45.145390 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800: Bootstrap starting.
I20260929 16:28:45.149634 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:45.151127 7220 log.cc:824] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:28:45 dist-test-slave-dssx krb5kdc[7130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:45.154526 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800: No bootstrap required, opened a new log
I20260929 16:28:45.161229 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.138368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.5.232.190:40169)
0929 16:28:45.139193 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.139205 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.139250 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:28:45.140930 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.140957 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.141531 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.141934 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:45.143840 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.145204 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.147054 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.147827 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.147977 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.153264 (+ 5287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:45.153273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:45.153290 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:45.153329 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:45.156638 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.157607 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.157614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.157621 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.157805 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.158222 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.158225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.158227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.158773 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:45.159005 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.159546 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.160002 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":382,"threads_started":1}
I20260929 16:28:45.163568 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea760c2817384d779a55a3250a0fe800" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40169 } }
I20260929 16:28:45.163942 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:45.164088 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea760c2817384d779a55a3250a0fe800, State: Initialized, Role: FOLLOWER
I20260929 16:28:45.164737 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea760c2817384d779a55a3250a0fe800" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40169 } }
I20260929 16:28:45.164953 7220 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:45.165048 7220 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:45.165189 7220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:45.167511 7220 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea760c2817384d779a55a3250a0fe800" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40169 } }
I20260929 16:28:45.168116 7220 leader_election.cc:302] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea760c2817384d779a55a3250a0fe800; no voters:
I20260929 16:28:45.168651 7220 leader_election.cc:288] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:28:45.168792 7225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:45.169526 7225 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [term 1 LEADER]: Becoming Leader. State: Replica: ea760c2817384d779a55a3250a0fe800, State: Running, Role: LEADER
I20260929 16:28:45.170150 7225 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea760c2817384d779a55a3250a0fe800" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40169 } }
I20260929 16:28:45.170933 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:45.173293 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea760c2817384d779a55a3250a0fe800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea760c2817384d779a55a3250a0fe800" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40169 } } }
I20260929 16:28:45.173835 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:45.173482 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea760c2817384d779a55a3250a0fe800. Latest consensus state: current_term: 1 leader_uuid: "ea760c2817384d779a55a3250a0fe800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea760c2817384d779a55a3250a0fe800" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40169 } } }
I20260929 16:28:45.174089 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:45.175721 7234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:45.181315 7234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:45.191143 7234 catalog_manager.cc:1347] Generated new cluster ID: 8153c08e49e6423197a8a9df79605a58
I20260929 16:28:45.191273 7234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:45.216264 7234 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:28:45.217757 7234 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:45.229030 7234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800: Generated new TSK 0
I20260929 16:28:45.230041 7234 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:28:45.281867 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40169
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:45.624614 7246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:45.624922 7246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:45.625114 7246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:45.632606 7246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:45.632735 7246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:45.632900 7246 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:28:45.633029 7246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:28:45.644797 7246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.232.190:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:45.646715 7246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:45.648128 7246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:45.662763 7252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.662830 7251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.662842 7254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:45.666920 7246 server_base.cc:1034] running on GCE node
I20260929 16:28:45.667613 7246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:45.668618 7246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:45.669881 7246 hybrid_clock.cc:648] HybridClock initialized: now 1790699325669815 us; error 42 us; skew 500 ppm
I20260929 16:28:45.670222 7246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:45 dist-test-slave-dssx krb5kdc[7130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:45.673794 7246 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:28:45.675570 7246 webserver.cc:460] Webserver started at http://127.5.232.129:34157/ using document root <none> and password file <none>
I20260929 16:28:45.676354 7246 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:45.676494 7246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:45.676834 7246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:45.679925 7246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "4a17cd7b76f041e68b01c91209d97e93"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-dssx"
server_key: "69b880733d3e8751bd9a7802a30923c3"
server_key_iv: "654742ee58f3ffc87dabd731ef0f565c"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.680775 7246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "4a17cd7b76f041e68b01c91209d97e93"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-dssx"
server_key: "69b880733d3e8751bd9a7802a30923c3"
server_key_iv: "654742ee58f3ffc87dabd731ef0f565c"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.686411 7246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:28:45.690373 7261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.691915 7246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:28:45.692119 7246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "4a17cd7b76f041e68b01c91209d97e93"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-dssx"
server_key: "69b880733d3e8751bd9a7802a30923c3"
server_key_iv: "654742ee58f3ffc87dabd731ef0f565c"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.692399 7246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:45.704083 7246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:45.708310 7246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:45.708611 7246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:45.711498 7246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:45.711616 7246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.711762 7246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:45.711858 7246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:28:45 dist-test-slave-dssx krb5kdc[7130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:45.739445 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.712349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:50257 (local address 127.5.232.190:40169)
0929 16:28:45.712803 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.712810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.714572 (+ 1762us) server_negotiation.cc:407] Connection header received
0929 16:28:45.716816 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.716825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.716924 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.717011 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:45.719755 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.721007 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.724591 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.725361 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.725521 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.731021 (+ 5500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:45.731033 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:45.731041 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:45.731104 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:45.734968 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.736508 (+ 1540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.736516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.736520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.736698 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.737334 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.737338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.737340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.737685 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:45.737864 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.738850 (+ 986us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.739091 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":1089,"threads_started":1}
I20260929 16:28:45.740577 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.713360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40169 (local address 127.5.232.129:50257)
0929 16:28:45.714353 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.714400 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.716406 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.717707 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.717717 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.718563 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:45.719553 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.719574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.721258 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.721267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.724324 (+ 3057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.724365 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.725700 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.725710 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.726054 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.726980 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:45.727007 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:45.730544 (+ 3537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:45.735212 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.735227 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.735260 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.736295 (+ 1035us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.736820 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.736828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.736835 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.737192 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.737956 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:45.737971 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:45.738456 (+ 485us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.738763 (+ 307us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.739372 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":378,"threads_started":1}
I20260929 16:28:45.763967 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.747527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40169 (local address 127.5.232.129:57565)
0929 16:28:45.748377 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.748404 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.748623 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.749010 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.749018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.749875 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:45.750458 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.750479 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.751853 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.751865 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.753707 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.753721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.754917 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.754923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.755062 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.755946 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:45.755964 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:45.756700 (+ 736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:45.760592 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.760598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.760619 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.761051 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.761535 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.761540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.761544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.761741 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.762449 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:45.762478 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:45.762706 (+ 228us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.762818 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.763269 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":443,"threads_started":1}
I20260929 16:28:45.764561 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.747518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:57565 (local address 127.5.232.190:40169)
0929 16:28:45.747749 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.747757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.748553 (+ 796us) server_negotiation.cc:407] Connection header received
0929 16:28:45.748769 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.748773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.748844 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.748938 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:45.750700 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.751623 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.753889 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.754445 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.754625 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.756924 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:45.756934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:45.756939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:45.757002 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:45.760420 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.761193 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.761200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.761207 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.761358 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.761909 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.761915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.761919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.762266 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:45.763936 (+ 1670us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.764062 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.764330 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:28:45.772254 7246 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:40683
I20260929 16:28:45.772298 7381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:40683 every 8 connection(s)
I20260929 16:28:45.773628 7246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:28:45.774320 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 7246
I20260929 16:28:45.774570 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:28:45.775388 6050 external_mini_cluster.cc:1428] Setting key 4392aa591714ad7b97b05228892309e9
I20260929 16:28:45.776252 7382 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40169
I20260929 16:28:45.776506 7382 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:45.777179 7382 heartbeater.cc:511] Master 127.5.232.190:40169 requested a full tablet report, sending...
I20260929 16:28:45.779492 7175 ts_manager.cc:194] Registered new tserver with Master: 4a17cd7b76f041e68b01c91209d97e93 (127.5.232.129:40683)
I20260929 16:28:45.781782 7175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:50257
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:28:45.818404 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40169
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:46.162365 7386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:46.162686 7386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:46.162859 7386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:46.171255 7386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:46.171401 7386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:46.171530 7386 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:28:46.171671 7386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:28:46.182220 7386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.232.190:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:46.183709 7386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:46.185079 7386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:46.200515 7392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:46.200441 7391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:46.200465 7394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:46.201180 7386 server_base.cc:1034] running on GCE node
I20260929 16:28:46.201993 7386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:46.202996 7386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:46.204246 7386 hybrid_clock.cc:648] HybridClock initialized: now 1790699326204187 us; error 29 us; skew 500 ppm
I20260929 16:28:46.204540 7386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:46 dist-test-slave-dssx krb5kdc[7130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:46.207855 7386 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:28:46.209870 7386 webserver.cc:460] Webserver started at http://127.5.232.130:36329/ using document root <none> and password file <none>
I20260929 16:28:46.210660 7386 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:46.210800 7386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:46.211135 7386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:46.214411 7386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "ddfd82116b28451dbc99f6432904493a"
format_stamp: "Formatted at 2026-09-29 16:28:46 on dist-test-slave-dssx"
server_key: "f1982362dbf1a3a4018d36d390c821dc"
server_key_iv: "d60b07a5795a59909b8efec0dede7ed9"
server_key_version: "encryptionkey@0"
I20260929 16:28:46.215306 7386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "ddfd82116b28451dbc99f6432904493a"
format_stamp: "Formatted at 2026-09-29 16:28:46 on dist-test-slave-dssx"
server_key: "f1982362dbf1a3a4018d36d390c821dc"
server_key_iv: "d60b07a5795a59909b8efec0dede7ed9"
server_key_version: "encryptionkey@0"
I20260929 16:28:46.220870 7386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:28:46.224759 7401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:46.226310 7386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:46.226500 7386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "ddfd82116b28451dbc99f6432904493a"
format_stamp: "Formatted at 2026-09-29 16:28:46 on dist-test-slave-dssx"
server_key: "f1982362dbf1a3a4018d36d390c821dc"
server_key_iv: "d60b07a5795a59909b8efec0dede7ed9"
server_key_version: "encryptionkey@0"
I20260929 16:28:46.226751 7386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:46.251098 7386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:46.255497 7386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:46.255805 7386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:46.258782 7386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:46.258882 7386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:46.259042 7386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:46.259145 7386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:28:46 dist-test-slave-dssx krb5kdc[7130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:46.282478 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.259283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:35129 (local address 127.5.232.190:40169)
0929 16:28:46.259540 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.259550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.261161 (+ 1611us) server_negotiation.cc:407] Connection header received
0929 16:28:46.263314 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.263324 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.263439 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.263558 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.266085 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.267448 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.270243 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.270770 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.270927 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.275512 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.275519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.275524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.275573 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.279065 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.279921 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.279932 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.279935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.280062 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.280547 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.280551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.280553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.280831 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.281000 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.281857 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.282112 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:28:46.284149 7410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.260043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40169 (local address 127.5.232.130:35129)
0929 16:28:46.260950 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.261001 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.262979 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.263840 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.263855 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.264826 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:46.265736 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.265760 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.267673 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.267682 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.269973 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.269998 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.271050 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.271062 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.271310 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.272119 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:46.272154 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:46.275027 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:46.279244 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.279258 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.279281 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.279769 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.280191 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.280196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.280200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.280413 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.280966 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:46.280976 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:46.281435 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.281769 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.282410 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":361,"threads_started":1}
I20260929 16:28:46.308283 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.291802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:52151 (local address 127.5.232.190:40169)
0929 16:28:46.292062 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.292070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.292992 (+ 922us) server_negotiation.cc:407] Connection header received
0929 16:28:46.293265 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.293274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.293372 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.293504 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.295074 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.295895 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.297672 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.298239 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.298396 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.301572 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.301578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.301582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.301616 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.304436 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.305637 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.305646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.305651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.305832 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.306387 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.306396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.306402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.306964 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.307154 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.307696 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.307933 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:28:46.308383 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.291803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40169 (local address 127.5.232.130:52151)
0929 16:28:46.292865 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.292882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.293069 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.293659 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.293666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.294521 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:46.294902 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.294913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.296065 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.296069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.297370 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.297446 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.298551 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.298570 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.298868 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.299734 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:46.299762 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:46.301272 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:46.304624 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.304633 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.304648 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.305459 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.305993 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.305999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.306003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.306227 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.307104 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:46.307111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:46.307394 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.307587 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.307952 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":703,"threads_started":1}
I20260929 16:28:46.323351 7386 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:40349
I20260929 16:28:46.323382 7520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:40349 every 8 connection(s)
I20260929 16:28:46.324810 7386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:28:46.327086 7521 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40169
I20260929 16:28:46.327363 7521 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:46.327565 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 7386
I20260929 16:28:46.327768 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:28:46.328150 7521 heartbeater.cc:511] Master 127.5.232.190:40169 requested a full tablet report, sending...
I20260929 16:28:46.328194 6050 external_mini_cluster.cc:1428] Setting key dbb20948f1db898e2ba71cf9bae20bf6
I20260929 16:28:46.329737 7174 ts_manager.cc:194] Registered new tserver with Master: ddfd82116b28451dbc99f6432904493a (127.5.232.130:40349)
I20260929 16:28:46.330945 7174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:35129
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:28:46.368705 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40169
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:38775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:46.701323 7525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:46.701742 7525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:46.701849 7525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:46.709339 7525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:46.709510 7525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:46.709596 7525 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:28:46.709679 7525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:28:46.719798 7525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:38775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.232.190:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:46.721204 7525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:46.722703 7525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:46.736825 7531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:46.736773 7533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:46.736796 7530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:46.737911 7525 server_base.cc:1034] running on GCE node
I20260929 16:28:46.738624 7525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:46.739570 7525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:46.740797 7525 hybrid_clock.cc:648] HybridClock initialized: now 1790699326740707 us; error 64 us; skew 500 ppm
I20260929 16:28:46.741120 7525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:46 dist-test-slave-dssx krb5kdc[7130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:46.744414 7525 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:28:46.746073 7525 webserver.cc:460] Webserver started at http://127.5.232.131:40011/ using document root <none> and password file <none>
I20260929 16:28:46.746830 7525 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:46.746966 7525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:46.747308 7525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:46.750411 7525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "57f651c446654ee09f7433c6c8d37b51"
format_stamp: "Formatted at 2026-09-29 16:28:46 on dist-test-slave-dssx"
server_key: "bf2280cccfc67b4883f7041a41db09d7"
server_key_iv: "1ccdecaee832ff714c462375f95f6541"
server_key_version: "encryptionkey@0"
I20260929 16:28:46.751253 7525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "57f651c446654ee09f7433c6c8d37b51"
format_stamp: "Formatted at 2026-09-29 16:28:46 on dist-test-slave-dssx"
server_key: "bf2280cccfc67b4883f7041a41db09d7"
server_key_iv: "1ccdecaee832ff714c462375f95f6541"
server_key_version: "encryptionkey@0"
I20260929 16:28:46.756736 7525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 16:28:46.760670 7540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:46.762502 7525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:46.762732 7525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "57f651c446654ee09f7433c6c8d37b51"
format_stamp: "Formatted at 2026-09-29 16:28:46 on dist-test-slave-dssx"
server_key: "bf2280cccfc67b4883f7041a41db09d7"
server_key_iv: "1ccdecaee832ff714c462375f95f6541"
server_key_version: "encryptionkey@0"
I20260929 16:28:46.762985 7525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:46.785910 7382 heartbeater.cc:503] Master 127.5.232.190:40169 was elected leader, sending a full tablet report...
I20260929 16:28:46.795172 7525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:46.799743 7525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:46.800033 7525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:46.802834 7525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:46.802953 7525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:46.803108 7525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:46.803188 7525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:28:46 dist-test-slave-dssx krb5kdc[7130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:46.831053 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.803478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:55245 (local address 127.5.232.190:40169)
0929 16:28:46.803982 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.803989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.805150 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 16:28:46.807397 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.807403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.807503 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.807632 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.810405 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.811750 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.814605 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.815444 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.815619 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.820919 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.820925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.820928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.820964 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.824687 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.826165 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.826171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.826175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.826334 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.827122 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.827131 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.827137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.827597 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.827790 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.830189 (+ 2399us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.830533 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356}
I20260929 16:28:46.833987 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.804035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40169 (local address 127.5.232.131:55245)
0929 16:28:46.804935 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.804996 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.807042 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.807934 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.807949 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.809009 (+ 1060us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:46.810088 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.810130 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.811991 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.811996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.814355 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.814402 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.815777 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.815787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.816103 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.817087 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:46.817143 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:46.820613 (+ 3470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:46.824993 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.825011 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.825035 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.825976 (+ 941us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.826508 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.826517 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.826523 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.826927 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.827895 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:46.827910 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:46.828629 (+ 719us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.829078 (+ 449us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.831728 (+ 2650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":398,"threads_started":1}
I20260929 16:28:46.850306 7525 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:42039
I20260929 16:28:46.850306 7659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:42039 every 8 connection(s)
I20260929 16:28:46.851821 7525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:28:46.854105 7660 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40169
I20260929 16:28:46.854393 7660 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:46.855100 7660 heartbeater.cc:511] Master 127.5.232.190:40169 requested a full tablet report, sending...
I20260929 16:28:46.856683 7174 ts_manager.cc:194] Registered new tserver with Master: 57f651c446654ee09f7433c6c8d37b51 (127.5.232.131:42039)
I20260929 16:28:46.856746 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.840996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:53407 (local address 127.5.232.190:40169)
0929 16:28:46.841246 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.841256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.841979 (+ 723us) server_negotiation.cc:407] Connection header received
0929 16:28:46.842290 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.842300 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.842416 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.842568 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.843597 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.845103 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.847311 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.847908 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.848030 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.850416 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.850430 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.850434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.850508 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.853775 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.854620 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.854626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.854630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.854778 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.855207 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.855212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.855218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.855494 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.855666 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.856079 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.856317 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:28:46.856766 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.840996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40169 (local address 127.5.232.131:53407)
0929 16:28:46.841826 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.841850 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.842071 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.842626 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.842629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.843047 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:46.843439 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.843456 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.845298 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.845302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.847038 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.847054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.848334 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.848339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.848474 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.849238 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:46.849276 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:46.850190 (+ 914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:46.853924 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.853930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.853934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.854484 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.854903 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.854909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.854912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.855059 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.855620 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:46.855625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:46.855817 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.855968 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.856311 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":438,"threads_started":1}
I20260929 16:28:46.858037 7174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:55245
I20260929 16:28:46.858521 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 7525
I20260929 16:28:46.858713 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:28:46.859216 6050 external_mini_cluster.cc:1428] Setting key 9508aae6e5ec5162a9dd2e306bf123fd
I20260929 16:28:46.862958 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:28:46 dist-test-slave-dssx krb5kdc[7130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:28:46 dist-test-slave-dssx krb5kdc[7130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:46.911198 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.896674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.5.232.190:40169)
0929 16:28:46.896858 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.896863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.897211 (+ 348us) server_negotiation.cc:407] Connection header received
0929 16:28:46.897444 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.897449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.897516 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.897616 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.899037 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.899770 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.901081 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.901600 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.901711 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.905833 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.905840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.905843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.905884 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.908477 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.909228 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.909232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.909234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.909356 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.909843 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.909846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.909848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.910129 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.910323 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.910621 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.910800 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:28:46.915896 7174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:28:46.922933 7174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:46.964149 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.936237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.5.232.130:40349)
0929 16:28:46.955355 (+ 19118us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.955361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.955390 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:28:46.955471 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.955477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.955696 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.955869 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:46.956797 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.958144 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.961265 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.962913 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.963057 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.963282 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.963659 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.963849 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18937,"thread_start_us":18802,"threads_started":1}
I20260929 16:28:46.964300 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.936182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:40349 (local address 127.0.0.1:58846)
0929 16:28:46.948086 (+ 11904us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.948166 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.948463 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.955942 (+ 7479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.955951 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.955994 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:46.956554 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.956565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.958326 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.958331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.961061 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.961073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.963125 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.963132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.963349 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.963413 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.963574 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.963783 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11454,"mutex_wait_us":55,"thread_start_us":12576,"threads_started":1}
I20260929 16:28:46.964936 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.936716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40683 (local address 127.0.0.1:40416)
0929 16:28:46.956494 (+ 19778us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.956522 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.956674 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.957688 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.957693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.957731 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:46.958154 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.958166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.959582 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.959593 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.961229 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.961246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.964057 (+ 2811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.964063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.964277 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.964316 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.964411 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.964549 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19615,"mutex_wait_us":5,"thread_start_us":15473,"threads_started":1}
I20260929 16:28:46.965040 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.937266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.5.232.131:42039)
0929 16:28:46.953094 (+ 15828us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.953107 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.953157 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:28:46.953311 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.953325 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.953776 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.954072 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:46.956779 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.958559 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.961485 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.962913 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.963057 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.963317 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.964516 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.964730 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15617,"thread_start_us":15411,"threads_started":1}
I20260929 16:28:46.965260 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.936543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:42039 (local address 127.0.0.1:44144)
0929 16:28:46.952569 (+ 16026us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.952597 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.952795 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.954554 (+ 1759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.954567 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.954616 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:46.956445 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.956466 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.958720 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.958725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.961056 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.961073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.963447 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.963452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.963618 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.963654 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.963756 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.963885 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15849,"mutex_wait_us":3,"thread_start_us":19593,"threads_started":1}
I20260929 16:28:46.965688 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.937268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.5.232.129:40683)
0929 16:28:46.951955 (+ 14687us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.951970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.956680 (+ 4710us) server_negotiation.cc:407] Connection header received
0929 16:28:46.956903 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.956912 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.957200 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.957445 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:46.958349 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.959423 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.961444 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.963559 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.963680 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.963888 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.965178 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.965364 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14444,"thread_start_us":14279,"threads_started":1}
I20260929 16:28:46.968823 7451 tablet_service.cc:1467] Processing CreateTablet for tablet 15093a8e84094f5aaa921363a77d63fc (DEFAULT_TABLE table=test-table [id=6d5b8681f14e492dae230d23468d6e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:46.968994 7312 tablet_service.cc:1467] Processing CreateTablet for tablet 15093a8e84094f5aaa921363a77d63fc (DEFAULT_TABLE table=test-table [id=6d5b8681f14e492dae230d23468d6e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:46.969229 7590 tablet_service.cc:1467] Processing CreateTablet for tablet 15093a8e84094f5aaa921363a77d63fc (DEFAULT_TABLE table=test-table [id=6d5b8681f14e492dae230d23468d6e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:46.970242 7312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15093a8e84094f5aaa921363a77d63fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:46.970291 7451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15093a8e84094f5aaa921363a77d63fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:46.970481 7590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15093a8e84094f5aaa921363a77d63fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:46.982455 7679 tablet_bootstrap.cc:492] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: Bootstrap starting.
I20260929 16:28:46.986696 7679 tablet_bootstrap.cc:654] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:46.987895 7679 log.cc:824] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:46.990568 7681 tablet_bootstrap.cc:492] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: Bootstrap starting.
I20260929 16:28:46.991221 7680 tablet_bootstrap.cc:492] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: Bootstrap starting.
I20260929 16:28:46.991303 7679 tablet_bootstrap.cc:492] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: No bootstrap required, opened a new log
I20260929 16:28:46.991604 7679 ts_tablet_manager.cc:1397] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260929 16:28:46.994508 7681 tablet_bootstrap.cc:654] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:46.994493 7680 tablet_bootstrap.cc:654] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:46.995838 7680 log.cc:824] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:46.996284 7681 log.cc:824] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:46.999190 7680 tablet_bootstrap.cc:492] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: No bootstrap required, opened a new log
I20260929 16:28:46.999501 7680 ts_tablet_manager.cc:1397] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 16:28:47.000586 7679 raft_consensus.cc:348] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } }
I20260929 16:28:47.001037 7679 raft_consensus.cc:374] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:47.001219 7681 tablet_bootstrap.cc:492] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: No bootstrap required, opened a new log
I20260929 16:28:47.001188 7679 raft_consensus.cc:729] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17cd7b76f041e68b01c91209d97e93, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.001513 7681 ts_tablet_manager.cc:1397] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 16:28:47.001868 7679 consensus_queue.cc:260] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } }
I20260929 16:28:47.003636 7679 ts_tablet_manager.cc:1428] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260929 16:28:47.008934 7680 raft_consensus.cc:348] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } }
I20260929 16:28:47.009366 7680 raft_consensus.cc:374] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:47.009509 7680 raft_consensus.cc:729] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfd82116b28451dbc99f6432904493a, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.010138 7680 consensus_queue.cc:260] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } }
I20260929 16:28:47.011220 7521 heartbeater.cc:503] Master 127.5.232.190:40169 was elected leader, sending a full tablet report...
I20260929 16:28:47.011230 7681 raft_consensus.cc:348] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } }
I20260929 16:28:47.011664 7681 raft_consensus.cc:374] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:47.011773 7681 raft_consensus.cc:729] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f651c446654ee09f7433c6c8d37b51, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.011869 7680 ts_tablet_manager.cc:1428] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260929 16:28:47.012354 7681 consensus_queue.cc:260] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } }
I20260929 16:28:47.013477 7660 heartbeater.cc:503] Master 127.5.232.190:40169 was elected leader, sending a full tablet report...
I20260929 16:28:47.013995 7681 ts_tablet_manager.cc:1428] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260929 16:28:47.026322 7383 tablet.cc:2369] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:28:47.076929 7522 tablet.cc:2369] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:28:47.103634 7661 tablet.cc:2369] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:28:47.106856 7687 raft_consensus.cc:482] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:47.107185 7687 raft_consensus.cc:504] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } }
I20260929 16:28:47.109338 7687 leader_election.cc:288] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddfd82116b28451dbc99f6432904493a (127.5.232.130:40349), 4a17cd7b76f041e68b01c91209d97e93 (127.5.232.129:40683)
I20260929 16:28:47.115281 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.109807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:52357 (local address 127.5.232.130:40349)
0929 16:28:47.109986 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.109992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.110180 (+ 188us) server_negotiation.cc:407] Connection header received
0929 16:28:47.110380 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.110386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.110495 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.110608 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:47.111003 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.111783 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.113405 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.114176 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.114518 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.114595 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.114867 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.115002 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:28:47.115284 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.109767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:40349 (local address 127.5.232.131:52357)
0929 16:28:47.110040 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.110063 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.110220 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.110647 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:47.110654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:47.110681 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:47.110861 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.110870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.111919 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.111927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.113222 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.113233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.114497 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.114504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.114707 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.114730 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:28:47.114848 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:28:47.114991 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 16:28:47.115594 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.109727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:55285 (local address 127.5.232.129:40683)
0929 16:28:47.109928 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.109937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.110595 (+ 658us) server_negotiation.cc:407] Connection header received
0929 16:28:47.110787 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.110796 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.110863 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.110968 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:47.111468 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.112153 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.113838 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.114698 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.114879 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.114950 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.115235 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.115358 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:28:47.115722 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.109819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40683 (local address 127.5.232.131:55285)
0929 16:28:47.110489 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.110502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.110640 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.110981 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:47.110985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:47.111007 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:47.111307 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.111315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.112298 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.112303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.113693 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.113703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.114904 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.114910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.115059 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.115080 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:28:47.115211 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 16:28:47.115340 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":1,"thread_start_us":323,"threads_started":1}
I20260929 16:28:47.116786 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15093a8e84094f5aaa921363a77d63fc" candidate_uuid: "57f651c446654ee09f7433c6c8d37b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfd82116b28451dbc99f6432904493a" is_pre_election: true
I20260929 16:28:47.116784 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15093a8e84094f5aaa921363a77d63fc" candidate_uuid: "57f651c446654ee09f7433c6c8d37b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17cd7b76f041e68b01c91209d97e93" is_pre_election: true
I20260929 16:28:47.117415 7476 raft_consensus.cc:2454] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f651c446654ee09f7433c6c8d37b51 in term 0.
I20260929 16:28:47.117415 7337 raft_consensus.cc:2454] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f651c446654ee09f7433c6c8d37b51 in term 0.
I20260929 16:28:47.118620 7542 leader_election.cc:302] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a17cd7b76f041e68b01c91209d97e93, 57f651c446654ee09f7433c6c8d37b51; no voters:
I20260929 16:28:47.119127 7687 raft_consensus.cc:2781] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:47.119302 7687 raft_consensus.cc:482] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:47.119429 7687 raft_consensus.cc:3037] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.121649 7687 raft_consensus.cc:504] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } }
I20260929 16:28:47.122299 7687 leader_election.cc:288] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [CANDIDATE]: Term 1 election: Requested vote from peers ddfd82116b28451dbc99f6432904493a (127.5.232.130:40349), 4a17cd7b76f041e68b01c91209d97e93 (127.5.232.129:40683)
I20260929 16:28:47.122907 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15093a8e84094f5aaa921363a77d63fc" candidate_uuid: "57f651c446654ee09f7433c6c8d37b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17cd7b76f041e68b01c91209d97e93"
I20260929 16:28:47.122902 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15093a8e84094f5aaa921363a77d63fc" candidate_uuid: "57f651c446654ee09f7433c6c8d37b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfd82116b28451dbc99f6432904493a"
I20260929 16:28:47.123122 7337 raft_consensus.cc:3037] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.123122 7476 raft_consensus.cc:3037] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.126319 7337 raft_consensus.cc:2454] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f651c446654ee09f7433c6c8d37b51 in term 1.
I20260929 16:28:47.126323 7476 raft_consensus.cc:2454] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f651c446654ee09f7433c6c8d37b51 in term 1.
I20260929 16:28:47.126951 7544 leader_election.cc:302] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f651c446654ee09f7433c6c8d37b51, ddfd82116b28451dbc99f6432904493a; no voters:
I20260929 16:28:47.127564 7687 raft_consensus.cc:2781] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:47.128489 7687 raft_consensus.cc:686] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 1 LEADER]: Becoming Leader. State: Replica: 57f651c446654ee09f7433c6c8d37b51, State: Running, Role: LEADER
I20260929 16:28:47.129128 7687 consensus_queue.cc:237] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } }
I20260929 16:28:47.134104 7175 catalog_manager.cc:5495] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57f651c446654ee09f7433c6c8d37b51 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "57f651c446654ee09f7433c6c8d37b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } health_report { overall_health: UNKNOWN } } }
I20260929 16:28:47.164201 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.158411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.5.232.131:42039)
0929 16:28:47.158630 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.158636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.158755 (+ 119us) server_negotiation.cc:407] Connection header received
0929 16:28:47.159122 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.159126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.159222 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.159379 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:47.159742 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.160443 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.161872 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.162304 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.162447 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.162981 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:47.163522 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:47.163690 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.163846 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.163930 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:28:47.172237 7337 raft_consensus.cc:1261] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Refusing update from remote peer 57f651c446654ee09f7433c6c8d37b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:47.172240 7476 raft_consensus.cc:1261] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [term 1 FOLLOWER]: Refusing update from remote peer 57f651c446654ee09f7433c6c8d37b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:47.173245 7689 consensus_queue.cc:1035] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 16:28:47.174098 7687 consensus_queue.cc:1035] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:47.183032 7692 mvcc.cc:205] Tried to move back new op lower bound from 7334704444087693312 to 7334704443925864448. Current Snapshot: MvccSnapshot[applied={T|T < 7334704444087693312}]
I20260929 16:28:47.186700 7693 mvcc.cc:205] Tried to move back new op lower bound from 7334704444087693312 to 7334704443925864448. Current Snapshot: MvccSnapshot[applied={T|T < 7334704444087693312}]
I20260929 16:28:47.191125 7694 mvcc.cc:205] Tried to move back new op lower bound from 7334704444087693312 to 7334704443925864448. Current Snapshot: MvccSnapshot[applied={T|T < 7334704444087693312}]
I20260929 16:28:47.204495 7174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:47.204814 7174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:47.208441 7174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800: Sending DeleteTablet for 3 replicas of tablet 15093a8e84094f5aaa921363a77d63fc
I20260929 16:28:47.209640 7451 tablet_service.cc:1514] Processing DeleteTablet for tablet 15093a8e84094f5aaa921363a77d63fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:47 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:58846
I20260929 16:28:47.209662 7312 tablet_service.cc:1514] Processing DeleteTablet for tablet 15093a8e84094f5aaa921363a77d63fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:47 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:40416
I20260929 16:28:47.209667 7590 tablet_service.cc:1514] Processing DeleteTablet for tablet 15093a8e84094f5aaa921363a77d63fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:47 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:44144
I20260929 16:28:47.215720 7175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:28:47.216658 7175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:47.228797 7311 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa63b53cb1849078bfe3b445f4027b0 (DEFAULT_TABLE table=test-table [id=82a2e78d42ee468da95783a5788bcb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:47.229097 7589 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa63b53cb1849078bfe3b445f4027b0 (DEFAULT_TABLE table=test-table [id=82a2e78d42ee468da95783a5788bcb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:47.229498 7311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa63b53cb1849078bfe3b445f4027b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:47.229763 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa63b53cb1849078bfe3b445f4027b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:47.234416 7450 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa63b53cb1849078bfe3b445f4027b0 (DEFAULT_TABLE table=test-table [id=82a2e78d42ee468da95783a5788bcb3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:47.235026 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa63b53cb1849078bfe3b445f4027b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:47.236470 7679 tablet_bootstrap.cc:492] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: Bootstrap starting.
I20260929 16:28:47.236521 7701 tablet_replica.cc:330] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: stopping tablet replica
I20260929 16:28:47.237165 7701 raft_consensus.cc:2229] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:47.238265 7701 raft_consensus.cc:2258] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:47.238408 7679 tablet_bootstrap.cc:654] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:47.240584 7701 ts_tablet_manager.cc:1905] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:47.247126 7681 tablet_bootstrap.cc:492] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: Bootstrap starting.
I20260929 16:28:47.248499 7701 ts_tablet_manager.cc:1918] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:28:47.248775 7701 log.cc:1196] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/15093a8e84094f5aaa921363a77d63fc
I20260929 16:28:47.249006 7681 tablet_bootstrap.cc:654] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:47.249548 7701 ts_tablet_manager.cc:1939] T 15093a8e84094f5aaa921363a77d63fc P 4a17cd7b76f041e68b01c91209d97e93: Deleting consensus metadata
I20260929 16:28:47.251442 7679 tablet_bootstrap.cc:492] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: No bootstrap required, opened a new log
I20260929 16:28:47.251502 7160 catalog_manager.cc:4841] TS 4a17cd7b76f041e68b01c91209d97e93 (127.5.232.129:40683): tablet 15093a8e84094f5aaa921363a77d63fc (table test-table [id=6d5b8681f14e492dae230d23468d6e96]) successfully deleted
I20260929 16:28:47.251590 7679 ts_tablet_manager.cc:1397] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20260929 16:28:47.252296 7679 raft_consensus.cc:348] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.252516 7679 raft_consensus.cc:374] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:47.252604 7679 raft_consensus.cc:729] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17cd7b76f041e68b01c91209d97e93, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.252923 7679 consensus_queue.cc:260] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.255153 7679 ts_tablet_manager.cc:1428] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 16:28:47.261690 7680 tablet_bootstrap.cc:492] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: Bootstrap starting.
I20260929 16:28:47.262305 7703 tablet_replica.cc:330] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: stopping tablet replica
I20260929 16:28:47.262821 7703 raft_consensus.cc:2229] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:47.263160 7702 tablet_replica.cc:330] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: stopping tablet replica
I20260929 16:28:47.263350 7703 raft_consensus.cc:2258] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:47.263507 7681 tablet_bootstrap.cc:492] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: No bootstrap required, opened a new log
I20260929 16:28:47.263593 7680 tablet_bootstrap.cc:654] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:47.263579 7702 raft_consensus.cc:2229] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:28:47.263669 7681 ts_tablet_manager.cc:1397] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20260929 16:28:47.264204 7702 raft_consensus.cc:2258] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:47.264937 7681 raft_consensus.cc:348] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.265190 7681 raft_consensus.cc:374] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:47.265300 7681 raft_consensus.cc:729] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f651c446654ee09f7433c6c8d37b51, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.265619 7681 consensus_queue.cc:260] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.266278 7681 ts_tablet_manager.cc:1428] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 16:28:47.267284 7680 tablet_bootstrap.cc:492] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: No bootstrap required, opened a new log
I20260929 16:28:47.267496 7680 ts_tablet_manager.cc:1397] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260929 16:28:47.268666 7703 ts_tablet_manager.cc:1905] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:47.268958 7680 raft_consensus.cc:348] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.269403 7680 raft_consensus.cc:374] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:47.269596 7680 raft_consensus.cc:729] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfd82116b28451dbc99f6432904493a, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.269972 7680 consensus_queue.cc:260] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.272946 7680 ts_tablet_manager.cc:1428] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 16:28:47.275581 7702 ts_tablet_manager.cc:1905] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:47.276293 7703 ts_tablet_manager.cc:1918] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:28:47.276460 7703 log.cc:1196] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/15093a8e84094f5aaa921363a77d63fc
I20260929 16:28:47.276998 7703 ts_tablet_manager.cc:1939] T 15093a8e84094f5aaa921363a77d63fc P ddfd82116b28451dbc99f6432904493a: Deleting consensus metadata
I20260929 16:28:47.278801 7162 catalog_manager.cc:4841] TS ddfd82116b28451dbc99f6432904493a (127.5.232.130:40349): tablet 15093a8e84094f5aaa921363a77d63fc (table test-table [id=6d5b8681f14e492dae230d23468d6e96]) successfully deleted
I20260929 16:28:47.282080 7702 ts_tablet_manager.cc:1918] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:28:47.282312 7702 log.cc:1196] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/15093a8e84094f5aaa921363a77d63fc
I20260929 16:28:47.282804 7702 ts_tablet_manager.cc:1939] T 15093a8e84094f5aaa921363a77d63fc P 57f651c446654ee09f7433c6c8d37b51: Deleting consensus metadata
I20260929 16:28:47.284381 7159 catalog_manager.cc:4841] TS 57f651c446654ee09f7433c6c8d37b51 (127.5.232.131:42039): tablet 15093a8e84094f5aaa921363a77d63fc (table test-table [id=6d5b8681f14e492dae230d23468d6e96]) successfully deleted
I20260929 16:28:47.478736 7686 raft_consensus.cc:482] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:47.479017 7686 raft_consensus.cc:504] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.480599 7686 leader_election.cc:288] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a17cd7b76f041e68b01c91209d97e93 (127.5.232.129:40683), 57f651c446654ee09f7433c6c8d37b51 (127.5.232.131:42039)
I20260929 16:28:47.489786 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.481746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:36705 (local address 127.5.232.129:40683)
0929 16:28:47.481939 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.481947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.482757 (+ 810us) server_negotiation.cc:407] Connection header received
0929 16:28:47.482913 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.482921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.482998 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.483101 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:47.483638 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.484358 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.487612 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.488555 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.488668 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.488719 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.489318 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.489486 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:28:47.489882 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.481280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40683 (local address 127.5.232.130:36705)
0929 16:28:47.482633 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.482652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.482764 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.483137 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:47.483144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:47.483174 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:47.483489 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.483497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.484492 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.484500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.487401 (+ 2901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.487419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.488883 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.488888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.489085 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.489123 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:28:47.489223 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:28:47.489471 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":196,"thread_start_us":1040,"threads_started":1}
I20260929 16:28:47.489984 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.481154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:60217 (local address 127.5.232.131:42039)
0929 16:28:47.481433 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.481444 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.481972 (+ 528us) server_negotiation.cc:407] Connection header received
0929 16:28:47.482326 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.482336 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.482428 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.482497 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:47.483280 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.484419 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.487577 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.488902 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.489054 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.489123 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.489542 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.489698 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:28:47.490640 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa63b53cb1849078bfe3b445f4027b0" candidate_uuid: "ddfd82116b28451dbc99f6432904493a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17cd7b76f041e68b01c91209d97e93" is_pre_election: true
I20260929 16:28:47.490844 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.481113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:42039 (local address 127.5.232.130:60217)
0929 16:28:47.481870 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.481890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.482090 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.482724 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:47.482728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:47.482774 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:47.483111 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.483122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.484555 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.484559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.487401 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.487419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.489110 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.489116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.489302 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.489330 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:28:47.490334 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0929 16:28:47.490477 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":188,"spinlock_wait_cycles":21888,"thread_start_us":820,"threads_started":1}
I20260929 16:28:47.490928 7337 raft_consensus.cc:2454] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfd82116b28451dbc99f6432904493a in term 0.
I20260929 16:28:47.491433 7403 leader_election.cc:302] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a17cd7b76f041e68b01c91209d97e93, ddfd82116b28451dbc99f6432904493a; no voters:
I20260929 16:28:47.491885 7686 raft_consensus.cc:2781] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:47.492022 7686 raft_consensus.cc:482] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:47.492117 7686 raft_consensus.cc:3037] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.491905 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa63b53cb1849078bfe3b445f4027b0" candidate_uuid: "ddfd82116b28451dbc99f6432904493a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f651c446654ee09f7433c6c8d37b51" is_pre_election: true
I20260929 16:28:47.492380 7610 raft_consensus.cc:2454] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfd82116b28451dbc99f6432904493a in term 0.
I20260929 16:28:47.495890 7686 raft_consensus.cc:504] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.496680 7686 leader_election.cc:288] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [CANDIDATE]: Term 1 election: Requested vote from peers 4a17cd7b76f041e68b01c91209d97e93 (127.5.232.129:40683), 57f651c446654ee09f7433c6c8d37b51 (127.5.232.131:42039)
I20260929 16:28:47.497144 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa63b53cb1849078bfe3b445f4027b0" candidate_uuid: "ddfd82116b28451dbc99f6432904493a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f651c446654ee09f7433c6c8d37b51"
I20260929 16:28:47.497196 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa63b53cb1849078bfe3b445f4027b0" candidate_uuid: "ddfd82116b28451dbc99f6432904493a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17cd7b76f041e68b01c91209d97e93"
I20260929 16:28:47.497362 7610 raft_consensus.cc:3037] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.497462 7337 raft_consensus.cc:3037] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.499706 7610 raft_consensus.cc:2454] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfd82116b28451dbc99f6432904493a in term 1.
I20260929 16:28:47.500265 7402 leader_election.cc:302] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f651c446654ee09f7433c6c8d37b51, ddfd82116b28451dbc99f6432904493a; no voters:
I20260929 16:28:47.500419 7337 raft_consensus.cc:2454] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfd82116b28451dbc99f6432904493a in term 1.
I20260929 16:28:47.500609 7686 raft_consensus.cc:2781] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:47.501130 7686 raft_consensus.cc:686] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 1 LEADER]: Becoming Leader. State: Replica: ddfd82116b28451dbc99f6432904493a, State: Running, Role: LEADER
I20260929 16:28:47.501642 7686 consensus_queue.cc:237] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.505333 7174 catalog_manager.cc:5495] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a reported cstate change: term changed from 0 to 1, leader changed from <none> to ddfd82116b28451dbc99f6432904493a (127.5.232.130). New cstate: current_term: 1 leader_uuid: "ddfd82116b28451dbc99f6432904493a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:28:47 dist-test-slave-dssx krb5kdc[7130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:47.592929 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.572059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40169 (local address 127.0.0.1:58854)
0929 16:28:47.572750 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.572780 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.572954 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.573911 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:47.573916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:47.574876 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:47.575129 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.575143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.576395 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.576400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.577992 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.578011 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.578743 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.578748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.578929 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.584424 (+ 5495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:47.584446 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:47.586936 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:47.590321 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:47.590337 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:47.590345 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:47.590770 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:47.591156 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:47.591161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:47.591165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:47.591302 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:47.591834 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:47.591843 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:47.592018 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 16:28:47.592194 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 16:28:47.592499 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":339,"threads_started":1}
I20260929 16:28:47.593010 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.572046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.5.232.190:40169)
0929 16:28:47.573580 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.573584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.573603 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:28:47.573650 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.573654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.573712 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.573806 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:47.575364 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.576244 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.578172 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.578561 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.578717 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.587200 (+ 8483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:47.587209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:47.587214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:47.587255 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:47.590169 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:47.590895 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:47.590902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:47.590908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:47.591035 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:47.591415 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:47.591420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:47.591423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:47.591696 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:47.591842 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.592285 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.592645 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":1434,"thread_start_us":1283,"threads_started":1}
I20260929 16:28:47.598909 7174 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:58854:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:28:47.600045 7174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:47.607642 7451 tablet_service.cc:1467] Processing CreateTablet for tablet 958906e65768439b84221f7bcaa799f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=091bc22ce9704976b34076e7a5b2d63a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:28:47.607592 7312 tablet_service.cc:1467] Processing CreateTablet for tablet 958906e65768439b84221f7bcaa799f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=091bc22ce9704976b34076e7a5b2d63a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:28:47.608219 7312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958906e65768439b84221f7bcaa799f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:47.608227 7451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958906e65768439b84221f7bcaa799f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:47.608091 7590 tablet_service.cc:1467] Processing CreateTablet for tablet 958906e65768439b84221f7bcaa799f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=091bc22ce9704976b34076e7a5b2d63a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:28:47.608713 7590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 958906e65768439b84221f7bcaa799f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:47.616443 7679 tablet_bootstrap.cc:492] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: Bootstrap starting.
I20260929 16:28:47.616443 7680 tablet_bootstrap.cc:492] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: Bootstrap starting.
I20260929 16:28:47.618283 7679 tablet_bootstrap.cc:654] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:47.618283 7680 tablet_bootstrap.cc:654] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:47.619010 7681 tablet_bootstrap.cc:492] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: Bootstrap starting.
I20260929 16:28:47.620879 7681 tablet_bootstrap.cc:654] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:47.628942 7681 tablet_bootstrap.cc:492] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: No bootstrap required, opened a new log
I20260929 16:28:47.629240 7681 ts_tablet_manager.cc:1397] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260929 16:28:47.629262 7679 tablet_bootstrap.cc:492] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: No bootstrap required, opened a new log
I20260929 16:28:47.629493 7679 ts_tablet_manager.cc:1397] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.002s
I20260929 16:28:47.630626 7679 raft_consensus.cc:348] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.631140 7679 raft_consensus.cc:374] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:47.630903 7681 raft_consensus.cc:348] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.631248 7679 raft_consensus.cc:729] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17cd7b76f041e68b01c91209d97e93, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.631371 7681 raft_consensus.cc:374] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:47.631563 7681 raft_consensus.cc:729] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f651c446654ee09f7433c6c8d37b51, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.631626 7679 consensus_queue.cc:260] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.632036 7681 consensus_queue.cc:260] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.632243 7685 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.632711 7685 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:47.632716 7689 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.633010 7689 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:47.634305 7679 ts_tablet_manager.cc:1428] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 16:28:47.635962 7680 tablet_bootstrap.cc:492] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: No bootstrap required, opened a new log
I20260929 16:28:47.636179 7680 ts_tablet_manager.cc:1397] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.012s
I20260929 16:28:47.636906 7680 raft_consensus.cc:348] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.637161 7680 raft_consensus.cc:374] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:47.637236 7680 raft_consensus.cc:729] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfd82116b28451dbc99f6432904493a, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.637530 7680 consensus_queue.cc:260] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.637955 7686 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.638259 7686 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:47.638839 7680 ts_tablet_manager.cc:1428] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 16:28:47.640512 7681 ts_tablet_manager.cc:1428] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260929 16:28:47.666819 7686 raft_consensus.cc:482] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:47.667048 7686 raft_consensus.cc:504] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.667728 7686 leader_election.cc:288] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a17cd7b76f041e68b01c91209d97e93 (127.5.232.129:40683), 57f651c446654ee09f7433c6c8d37b51 (127.5.232.131:42039)
I20260929 16:28:47.668335 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958906e65768439b84221f7bcaa799f4" candidate_uuid: "ddfd82116b28451dbc99f6432904493a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17cd7b76f041e68b01c91209d97e93" is_pre_election: true
I20260929 16:28:47.668411 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958906e65768439b84221f7bcaa799f4" candidate_uuid: "ddfd82116b28451dbc99f6432904493a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f651c446654ee09f7433c6c8d37b51" is_pre_election: true
I20260929 16:28:47.668637 7337 raft_consensus.cc:2454] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfd82116b28451dbc99f6432904493a in term 0.
I20260929 16:28:47.668707 7610 raft_consensus.cc:2454] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfd82116b28451dbc99f6432904493a in term 0.
I20260929 16:28:47.669081 7403 leader_election.cc:302] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a17cd7b76f041e68b01c91209d97e93, ddfd82116b28451dbc99f6432904493a; no voters:
I20260929 16:28:47.669365 7686 raft_consensus.cc:2781] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:47.669492 7686 raft_consensus.cc:482] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:47.669601 7686 raft_consensus.cc:3037] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.672004 7686 raft_consensus.cc:504] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.672717 7686 leader_election.cc:288] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [CANDIDATE]: Term 1 election: Requested vote from peers 4a17cd7b76f041e68b01c91209d97e93 (127.5.232.129:40683), 57f651c446654ee09f7433c6c8d37b51 (127.5.232.131:42039)
I20260929 16:28:47.673144 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958906e65768439b84221f7bcaa799f4" candidate_uuid: "ddfd82116b28451dbc99f6432904493a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17cd7b76f041e68b01c91209d97e93"
I20260929 16:28:47.673436 7337 raft_consensus.cc:3037] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.673591 7610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "958906e65768439b84221f7bcaa799f4" candidate_uuid: "ddfd82116b28451dbc99f6432904493a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f651c446654ee09f7433c6c8d37b51"
I20260929 16:28:47.673818 7610 raft_consensus.cc:3037] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.675832 7337 raft_consensus.cc:2454] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfd82116b28451dbc99f6432904493a in term 1.
I20260929 16:28:47.676294 7403 leader_election.cc:302] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a17cd7b76f041e68b01c91209d97e93, ddfd82116b28451dbc99f6432904493a; no voters:
I20260929 16:28:47.676558 7686 raft_consensus.cc:2781] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:47.676759 7686 raft_consensus.cc:686] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [term 1 LEADER]: Becoming Leader. State: Replica: ddfd82116b28451dbc99f6432904493a, State: Running, Role: LEADER
I20260929 16:28:47.676856 7610 raft_consensus.cc:2454] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfd82116b28451dbc99f6432904493a in term 1.
I20260929 16:28:47.677006 7686 consensus_queue.cc:237] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } }
I20260929 16:28:47.678716 7709 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: TxnStatusTablet state changed. Reason: New leader ddfd82116b28451dbc99f6432904493a. Latest consensus state: current_term: 1 leader_uuid: "ddfd82116b28451dbc99f6432904493a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.679020 7709 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:28:47.679656 7722 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:28:47.679912 7722 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:28:47.681483 7175 catalog_manager.cc:5495] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a reported cstate change: term changed from 0 to 1, leader changed from <none> to ddfd82116b28451dbc99f6432904493a (127.5.232.130). New cstate: current_term: 1 leader_uuid: "ddfd82116b28451dbc99f6432904493a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } health_report { overall_health: UNKNOWN } } }
I20260929 16:28:47.710121 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.703379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:40349 (local address 127.0.0.1:58856)
0929 16:28:47.703694 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.703722 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.703883 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.704464 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:47.704467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:47.704476 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:28:47.704618 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.704624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.706067 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.706071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.707066 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.707077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.708255 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.708260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.708434 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.708503 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:28:47.709669 (+ 1166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:28:47.709687 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:28:47.709753 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 16:28:47.709876 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:28:47.710285 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.703365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.5.232.130:40349)
0929 16:28:47.704037 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.704044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.704071 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:28:47.704168 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.704174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.704274 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.704445 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:47.704768 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.705942 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.707222 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.708058 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.708203 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.708703 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:47.709548 (+ 845us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:47.709694 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.709802 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.709899 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":387,"threads_started":1}
I20260929 16:28:47.714708 7610 raft_consensus.cc:1261] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51 [term 1 FOLLOWER]: Refusing update from remote peer ddfd82116b28451dbc99f6432904493a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:47.714993 7337 raft_consensus.cc:1261] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Refusing update from remote peer ddfd82116b28451dbc99f6432904493a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:47.715713 7686 consensus_queue.cc:1035] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:47.716343 7686 consensus_queue.cc:1035] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:47.732472 7689 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: TxnStatusTablet state changed. Reason: New leader ddfd82116b28451dbc99f6432904493a. Latest consensus state: current_term: 1 leader_uuid: "ddfd82116b28451dbc99f6432904493a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.732846 7689 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:47.737686 7709 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddfd82116b28451dbc99f6432904493a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.737969 7709 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:28:47.742214 7685 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: TxnStatusTablet state changed. Reason: New leader ddfd82116b28451dbc99f6432904493a. Latest consensus state: current_term: 1 leader_uuid: "ddfd82116b28451dbc99f6432904493a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.742571 7685 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:47.743963 7689 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddfd82116b28451dbc99f6432904493a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.744212 7689 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P 57f651c446654ee09f7433c6c8d37b51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:47.746027 7686 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddfd82116b28451dbc99f6432904493a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.746294 7686 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P ddfd82116b28451dbc99f6432904493a: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:28:47.751482 7685 tablet_replica.cc:439] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddfd82116b28451dbc99f6432904493a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 } } peers { permanent_uuid: "ddfd82116b28451dbc99f6432904493a" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 40349 } } peers { permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 } } }
I20260929 16:28:47.751751 7685 tablet_replica.cc:442] T 958906e65768439b84221f7bcaa799f4 P 4a17cd7b76f041e68b01c91209d97e93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:47.755615 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.746569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.5.232.130:40349)
0929 16:28:47.746716 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.746722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.747303 (+ 581us) server_negotiation.cc:407] Connection header received
0929 16:28:47.747573 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.747578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.747654 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.747714 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:47.748867 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.750463 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.753676 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.754328 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.754475 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.754888 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:47.754998 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:47.755104 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.755220 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.755311 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:28:47.773140 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.766322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:58001 (local address 127.5.232.130:40349)
0929 16:28:47.766535 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.766556 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.767146 (+ 590us) server_negotiation.cc:407] Connection header received
0929 16:28:47.767354 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.767358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.767453 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.767560 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:47.768218 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.769004 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.771246 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.771694 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.771858 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.772395 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:47.772492 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:47.772586 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.772723 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.772795 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 16:28:47.773238 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.766456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:40349 (local address 127.5.232.130:58001)
0929 16:28:47.767039 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.767058 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.767229 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.767614 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:47.767619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:47.767633 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:28:47.768012 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.768025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.769127 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.769133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.771088 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.771103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.771926 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.771932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.772154 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.772213 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:28:47.772597 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:28:47.772615 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:28:47.772708 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 16:28:47.772830 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":320,"threads_started":1}
I20260929 16:28:47.784958 7337 raft_consensus.cc:1261] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Refusing update from remote peer ddfd82116b28451dbc99f6432904493a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:47.784951 7610 raft_consensus.cc:1261] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [term 1 FOLLOWER]: Refusing update from remote peer ddfd82116b28451dbc99f6432904493a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:47.785607 7686 consensus_queue.cc:1035] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a17cd7b76f041e68b01c91209d97e93" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:47.785852 7709 consensus_queue.cc:1035] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f651c446654ee09f7433c6c8d37b51" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:47.871937 7175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:47.872192 7175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:47.875864 7175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea760c2817384d779a55a3250a0fe800: Sending DeleteTablet for 3 replicas of tablet 1aa63b53cb1849078bfe3b445f4027b0
I20260929 16:28:47.876729 7590 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aa63b53cb1849078bfe3b445f4027b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:47 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:44144
I20260929 16:28:47.876753 7312 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aa63b53cb1849078bfe3b445f4027b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:47 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:40416
I20260929 16:28:47.877748 7450 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aa63b53cb1849078bfe3b445f4027b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:47 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:58846
I20260929 16:28:47.899839 7743 tablet_replica.cc:330] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: stopping tablet replica
I20260929 16:28:47.900096 7743 raft_consensus.cc:2229] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:47.900247 7745 tablet_replica.cc:330] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: stopping tablet replica
I20260929 16:28:47.900380 7743 raft_consensus.cc:2258] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:47.900594 7745 raft_consensus.cc:2229] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:28:47.901467 7745 raft_consensus.cc:2258] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:47.901710 7743 ts_tablet_manager.cc:1905] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:47.903343 7745 ts_tablet_manager.cc:1905] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:47.913825 7744 tablet_replica.cc:330] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: stopping tablet replica
I20260929 16:28:47.914062 7744 raft_consensus.cc:2229] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:47.914358 7744 raft_consensus.cc:2258] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:47.918213 7744 ts_tablet_manager.cc:1905] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:47.918347 7743 ts_tablet_manager.cc:1918] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:28:47.918505 7743 log.cc:1196] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/1aa63b53cb1849078bfe3b445f4027b0
I20260929 16:28:47.918787 7745 ts_tablet_manager.cc:1918] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:28:47.919392 7745 log.cc:1196] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/1aa63b53cb1849078bfe3b445f4027b0
I20260929 16:28:47.919394 7743 ts_tablet_manager.cc:1939] T 1aa63b53cb1849078bfe3b445f4027b0 P 57f651c446654ee09f7433c6c8d37b51: Deleting consensus metadata
I20260929 16:28:47.919855 7745 ts_tablet_manager.cc:1939] T 1aa63b53cb1849078bfe3b445f4027b0 P ddfd82116b28451dbc99f6432904493a: Deleting consensus metadata
I20260929 16:28:47.920607 7159 catalog_manager.cc:4841] TS 57f651c446654ee09f7433c6c8d37b51 (127.5.232.131:42039): tablet 1aa63b53cb1849078bfe3b445f4027b0 (table test-table [id=82a2e78d42ee468da95783a5788bcb3f]) successfully deleted
I20260929 16:28:47.920879 7162 catalog_manager.cc:4841] TS ddfd82116b28451dbc99f6432904493a (127.5.232.130:40349): tablet 1aa63b53cb1849078bfe3b445f4027b0 (table test-table [id=82a2e78d42ee468da95783a5788bcb3f]) successfully deleted
Sep 29 16:28:47 dist-test-slave-dssx krb5kdc[7130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:28:47.927366 7744 ts_tablet_manager.cc:1918] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:28:47.927558 7744 log.cc:1196] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/1aa63b53cb1849078bfe3b445f4027b0
I20260929 16:28:47.927987 7744 ts_tablet_manager.cc:1939] T 1aa63b53cb1849078bfe3b445f4027b0 P 4a17cd7b76f041e68b01c91209d97e93: Deleting consensus metadata
I20260929 16:28:47.928983 7160 catalog_manager.cc:4841] TS 4a17cd7b76f041e68b01c91209d97e93 (127.5.232.129:40683): tablet 1aa63b53cb1849078bfe3b445f4027b0 (table test-table [id=82a2e78d42ee468da95783a5788bcb3f]) successfully deleted
I20260929 16:28:47.933151 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.898698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.5.232.129:40683)
0929 16:28:47.898955 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.898965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.902277 (+ 3312us) server_negotiation.cc:407] Connection header received
0929 16:28:47.903793 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.903800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.903887 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.903960 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:47.909740 (+ 5780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.910963 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.918544 (+ 7581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.919225 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.919362 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.925032 (+ 5670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:47.925047 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:47.925068 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:47.925124 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:47.929335 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:47.930146 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:47.930158 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:47.930162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:47.930332 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:47.930796 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:47.930802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:47.930806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:47.931286 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:47.931588 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.931926 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.932777 (+ 851us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 16:28:47.933951 7377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40432
I20260929 16:28:47.951256 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.936730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.5.232.190:40169)
0929 16:28:47.936959 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.936965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.937229 (+ 264us) server_negotiation.cc:407] Connection header received
0929 16:28:47.937430 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.937436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.937515 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.937619 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:47.938853 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.939988 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.941169 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.941894 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.942023 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.944991 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:47.944999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:47.945004 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:47.945050 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:47.948360 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:47.949071 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:47.949077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:47.949081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:47.949222 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:47.949682 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:47.949687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:47.949691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:47.950075 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:47.950315 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.950668 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.950920 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 16:28:47.951892 7175 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58862
I20260929 16:28:47.953979 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7246
I20260929 16:28:47.969739 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.954838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.5.232.129:40683)
0929 16:28:47.955027 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.955033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.956646 (+ 1613us) server_negotiation.cc:407] Connection header received
0929 16:28:47.956744 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.956750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.956832 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.956942 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:47.958153 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.959334 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.960476 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.961123 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.961253 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.963545 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:47.963552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:47.963557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:47.963600 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:47.966926 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:47.967629 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:47.967635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:47.967639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:47.967796 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:47.968145 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:47.968151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:47.968154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:47.968545 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:47.968887 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.969153 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.969366 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 16:28:47.970424 7377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40444
W20260929 16:28:47.971050 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7246: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:28:47.971236 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 7246
I20260929 16:28:47.989166 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7386
Sep 29 16:28:47 dist-test-slave-dssx krb5kdc[7130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:28:48.007666 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.990008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.5.232.130:40349)
0929 16:28:47.990222 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.990234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.990354 (+ 120us) server_negotiation.cc:407] Connection header received
0929 16:28:47.990616 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.990620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.990682 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.990766 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:47.992215 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.992918 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.994702 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.995168 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.995272 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:48.000252 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:48.000267 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:48.000288 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:48.000337 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:48.004520 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:48.005317 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:48.005333 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:48.005337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:48.005558 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:48.005947 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:48.005952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:48.005956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:48.006450 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:48.006837 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:48.007015 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 16:28:48.007270 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:28:48.008479 7516 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58868
W20260929 16:28:48.009194 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7386: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:28:48.009447 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 7386
I20260929 16:28:48.028789 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7525
Sep 29 16:28:48 dist-test-slave-dssx krb5kdc[7130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:28:48.044162 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:48.029585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.5.232.131:42039)
0929 16:28:48.030078 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:48.030083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:48.030101 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:28:48.030165 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:48.030169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:48.030244 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:48.030354 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:48.031558 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:48.032277 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:48.033575 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:48.034007 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:48.034114 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:48.038588 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:48.038605 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:48.038631 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:48.038673 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:48.041536 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:48.042282 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:48.042289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:48.042291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:48.042418 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:48.042807 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:48.042811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:48.042813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:48.043192 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:48.043467 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:48.043727 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:28:48.043911 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":283,"threads_started":1}
W20260929 16:28:48.044930 7655 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44170
W20260929 16:28:48.045706 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7525: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:28:48.045851 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 7525
I20260929 16:28:48.066658 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7143
I20260929 16:28:48.067823 7215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:48.184109 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 7143
2026-09-29T16:28:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4765 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:28:48 dist-test-slave-dssx krb5kdc[7755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:28:48 dist-test-slave-dssx krb5kdc[7755](info): set up 2 sockets
Sep 29 16:28:48 dist-test-slave-dssx krb5kdc[7755](info): commencing operation
krb5kdc: starting...
W20260929 16:28:49.265720 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:28:49 dist-test-slave-dssx krb5kdc[7755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:28:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:28:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:28:49.454505 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:36861
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:36861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:49.791062 7768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:49.791375 7768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:49.791546 7768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:49.798940 7768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:49.799067 7768 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:28:49.799142 7768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:49.799237 7768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:49.799322 7768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:49.799437 7768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:49.809651 7768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:36861
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:36861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:49.811015 7768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:49.812377 7768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:49.824476 7774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:49.824527 7773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:49.824435 7776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:49.827016 7768 server_base.cc:1034] running on GCE node
I20260929 16:28:49.827845 7768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:49.829336 7768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:49.830606 7768 hybrid_clock.cc:648] HybridClock initialized: now 1790699329830547 us; error 37 us; skew 500 ppm
I20260929 16:28:49.830950 7768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:49 dist-test-slave-dssx krb5kdc[7755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699329, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:49.834182 7768 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:28:49.835754 7768 webserver.cc:460] Webserver started at http://127.5.232.190:38593/ using document root <none> and password file <none>
I20260929 16:28:49.836514 7768 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:49.836647 7768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:49.836961 7768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:49.839998 7768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "deabc511b35a452e9d2834006cbd5018"
format_stamp: "Formatted at 2026-09-29 16:28:49 on dist-test-slave-dssx"
server_key: "042935d9c2204be9192d798676231144"
server_key_iv: "89d2e24c05810674907588cbe6172c5c"
server_key_version: "encryptionkey@0"
I20260929 16:28:49.840791 7768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "deabc511b35a452e9d2834006cbd5018"
format_stamp: "Formatted at 2026-09-29 16:28:49 on dist-test-slave-dssx"
server_key: "042935d9c2204be9192d798676231144"
server_key_iv: "89d2e24c05810674907588cbe6172c5c"
server_key_version: "encryptionkey@0"
I20260929 16:28:49.846143 7768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:28:49.849700 7783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:49.851049 7768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:28:49.851243 7768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "deabc511b35a452e9d2834006cbd5018"
format_stamp: "Formatted at 2026-09-29 16:28:49 on dist-test-slave-dssx"
server_key: "042935d9c2204be9192d798676231144"
server_key_iv: "89d2e24c05810674907588cbe6172c5c"
server_key_version: "encryptionkey@0"
I20260929 16:28:49.851480 7768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:49.886029 7768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:49.892187 7768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:49.892519 7768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:49.918670 7768 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:36861
I20260929 16:28:49.918687 7844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:36861 every 8 connection(s)
I20260929 16:28:49.920083 7768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:49.923659 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 7768
I20260929 16:28:49.923878 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:28:49.924455 7845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:49.924691 6050 external_mini_cluster.cc:1428] Setting key 2e031ff3e80a61c3330753ac5c093b6e
I20260929 16:28:49.939955 7845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018: Bootstrap starting.
Sep 29 16:28:49 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:49.943451 7845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:49.944504 7845 log.cc:824] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:49.947386 7845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018: No bootstrap required, opened a new log
I20260929 16:28:49.948721 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:49.926830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.5.232.190:36861)
0929 16:28:49.927755 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:49.927772 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:49.927829 (+ 57us) server_negotiation.cc:407] Connection header received
0929 16:28:49.929961 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:49.929983 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:49.930395 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:49.930747 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:49.932558 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.933923 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.935702 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.936412 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.936552 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:49.941287 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:49.941297 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:49.941315 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:49.941359 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:49.944533 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:49.945485 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:49.945492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:49.945498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:49.945665 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:49.946048 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:49.946055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:49.946058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:49.946577 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:49.946868 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:49.947296 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 16:28:49.947745 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":447,"threads_started":1}
I20260929 16:28:49.955734 7845 raft_consensus.cc:348] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deabc511b35a452e9d2834006cbd5018" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36861 } }
I20260929 16:28:49.956158 7845 raft_consensus.cc:374] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:49.956243 7845 raft_consensus.cc:729] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deabc511b35a452e9d2834006cbd5018, State: Initialized, Role: FOLLOWER
I20260929 16:28:49.956853 7845 consensus_queue.cc:260] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deabc511b35a452e9d2834006cbd5018" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36861 } }
I20260929 16:28:49.957077 7845 raft_consensus.cc:388] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:49.957202 7845 raft_consensus.cc:482] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:49.957376 7845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:49.959857 7845 raft_consensus.cc:504] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deabc511b35a452e9d2834006cbd5018" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36861 } }
I20260929 16:28:49.960428 7845 leader_election.cc:302] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deabc511b35a452e9d2834006cbd5018; no voters:
I20260929 16:28:49.961082 7845 leader_election.cc:288] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:28:49.961218 7850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:49.961923 7850 raft_consensus.cc:686] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [term 1 LEADER]: Becoming Leader. State: Replica: deabc511b35a452e9d2834006cbd5018, State: Running, Role: LEADER
I20260929 16:28:49.962447 7850 consensus_queue.cc:237] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deabc511b35a452e9d2834006cbd5018" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36861 } }
I20260929 16:28:49.963176 7845 sys_catalog.cc:564] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:49.965265 7851 sys_catalog.cc:455] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deabc511b35a452e9d2834006cbd5018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deabc511b35a452e9d2834006cbd5018" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36861 } } }
I20260929 16:28:49.965763 7851 sys_catalog.cc:458] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:49.966290 7852 sys_catalog.cc:455] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deabc511b35a452e9d2834006cbd5018. Latest consensus state: current_term: 1 leader_uuid: "deabc511b35a452e9d2834006cbd5018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deabc511b35a452e9d2834006cbd5018" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36861 } } }
I20260929 16:28:49.966614 7852 sys_catalog.cc:458] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:49.970574 7858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:49.976262 7858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:49.985857 7858 catalog_manager.cc:1347] Generated new cluster ID: 658d3c296bce477b94adfb88206af5e9
I20260929 16:28:49.985994 7858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:50.002242 7858 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:28:50.003561 7858 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:50.013964 7858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018: Generated new TSK 0
I20260929 16:28:50.014824 7858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:28:50.070842 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:36861
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:50.404465 7871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:50.404770 7871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:50.404955 7871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:50.412389 7871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:50.412523 7871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:50.412681 7871 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:28:50.412772 7871 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:28:50.412918 7871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:28:50.423367 7871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.232.190:36861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:50.424710 7871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:50.426123 7871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:50.439354 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:50.439409 7879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:50.439299 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:50.440500 7871 server_base.cc:1034] running on GCE node
I20260929 16:28:50.441095 7871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:50.442109 7871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:50.443311 7871 hybrid_clock.cc:648] HybridClock initialized: now 1790699330443240 us; error 62 us; skew 500 ppm
I20260929 16:28:50.443656 7871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:50 dist-test-slave-dssx krb5kdc[7755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:50.446938 7871 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:28:50.448621 7871 webserver.cc:460] Webserver started at http://127.5.232.129:35883/ using document root <none> and password file <none>
I20260929 16:28:50.449359 7871 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:50.449565 7871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:50.449896 7871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:50.452836 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "9d2243e6db114eda9fb2fbe25c468b17"
format_stamp: "Formatted at 2026-09-29 16:28:50 on dist-test-slave-dssx"
server_key: "4036b66c6684d8858c3223c4cb77380a"
server_key_iv: "305a8ffe6beaaa85105146c8a39c2821"
server_key_version: "encryptionkey@0"
I20260929 16:28:50.453696 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "9d2243e6db114eda9fb2fbe25c468b17"
format_stamp: "Formatted at 2026-09-29 16:28:50 on dist-test-slave-dssx"
server_key: "4036b66c6684d8858c3223c4cb77380a"
server_key_iv: "305a8ffe6beaaa85105146c8a39c2821"
server_key_version: "encryptionkey@0"
I20260929 16:28:50.459201 7871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:28:50.463065 7886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:50.464576 7871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:28:50.464764 7871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "9d2243e6db114eda9fb2fbe25c468b17"
format_stamp: "Formatted at 2026-09-29 16:28:50 on dist-test-slave-dssx"
server_key: "4036b66c6684d8858c3223c4cb77380a"
server_key_iv: "305a8ffe6beaaa85105146c8a39c2821"
server_key_version: "encryptionkey@0"
I20260929 16:28:50.464977 7871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:50.480827 7871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:50.484858 7871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:50.485164 7871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:50.487815 7871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:50.487977 7871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:50.488116 7871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:50.488211 7871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:28:50 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:50.512621 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:50.488653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:51743 (local address 127.5.232.190:36861)
0929 16:28:50.489107 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:50.489113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:50.490499 (+ 1386us) server_negotiation.cc:407] Connection header received
0929 16:28:50.493614 (+ 3115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:50.493621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:50.493722 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:50.493859 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:50.496673 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:50.497533 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:50.500119 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:50.500688 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:50.500827 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:50.504932 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:50.504941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:50.504951 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:50.504984 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:50.508863 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:50.509744 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:50.509753 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:50.509759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:50.509941 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:50.510545 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:50.510551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:50.510554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:50.510985 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:50.511133 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:50.512029 (+ 896us) server_negotiation.cc:299] Negotiation successful
0929 16:28:50.512251 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260929 16:28:50.514101 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:50.489303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36861 (local address 127.5.232.129:51743)
0929 16:28:50.490292 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:50.490344 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:50.493210 (+ 2866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:50.494185 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:50.494202 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:50.495284 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:50.496438 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:50.496474 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:50.497692 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:50.497727 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:50.499870 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:50.499882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:50.500958 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:50.500965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:50.501225 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:50.502029 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:50.502055 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:50.504691 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:50.509047 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:50.509054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:50.509069 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:50.509614 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:50.510072 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:50.510079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:50.510085 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:50.510408 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:50.511206 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:50.511212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:50.511628 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 16:28:50.511929 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 16:28:50.512547 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":373,"threads_started":1}
I20260929 16:28:50.538010 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:50.520783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:33199 (local address 127.5.232.190:36861)
0929 16:28:50.520973 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:50.520981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:50.521713 (+ 732us) server_negotiation.cc:407] Connection header received
0929 16:28:50.521973 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:50.521978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:50.522061 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:50.522145 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:50.524166 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:50.525113 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:50.527365 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:50.527918 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:50.528092 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:50.530807 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:50.530818 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:50.530826 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:50.530879 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:50.534890 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:50.535708 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:50.535713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:50.535716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:50.535878 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:50.536293 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:50.536299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:50.536302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:50.536713 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:50.536902 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:50.537275 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 16:28:50.537604 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:28:50.538106 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:50.520804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36861 (local address 127.5.232.129:33199)
0929 16:28:50.521584 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:50.521611 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:50.521777 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:50.522407 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:50.522414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:50.523265 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:50.523911 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:50.523932 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:50.525265 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:50.525272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:50.527198 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:50.527221 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:50.528339 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:50.528344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:50.528570 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:50.529346 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:50.529410 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:50.530556 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:50.535076 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:50.535083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:50.535092 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:50.535569 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:50.535990 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:50.535993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:50.535995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:50.536152 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:50.536840 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:50.536843 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:50.536997 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 16:28:50.537109 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:28:50.537425 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":354,"threads_started":1}
I20260929 16:28:50.538115 7871 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:46825
I20260929 16:28:50.538316 8006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:46825 every 8 connection(s)
I20260929 16:28:50.539824 7871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:28:50.541710 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 7871
I20260929 16:28:50.541934 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:28:50.542744 6050 external_mini_cluster.cc:1428] Setting key 6a1c9c464caef2afa61809eee15d1220
I20260929 16:28:50.543296 8007 heartbeater.cc:347] Connected to a master server at 127.5.232.190:36861
I20260929 16:28:50.543573 8007 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:50.544375 8007 heartbeater.cc:511] Master 127.5.232.190:36861 requested a full tablet report, sending...
I20260929 16:28:50.546825 7800 ts_manager.cc:194] Registered new tserver with Master: 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825)
I20260929 16:28:50.549063 7800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:51743
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:28:50.584367 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:36861
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:50.918673 8011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:50.918957 8011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:50.919126 8011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:50.926790 8011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:50.926909 8011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:50.927078 8011 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:28:50.927158 8011 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:28:50.927295 8011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:28:50.938130 8011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.232.190:36861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:50.939518 8011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:50.940857 8011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:50.954203 8016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:50.954229 8019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:50.954164 8017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:50.955060 8011 server_base.cc:1034] running on GCE node
I20260929 16:28:50.955771 8011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:50.956736 8011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:50.957935 8011 hybrid_clock.cc:648] HybridClock initialized: now 1790699330957857 us; error 79 us; skew 500 ppm
I20260929 16:28:50.958266 8011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:50 dist-test-slave-dssx krb5kdc[7755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:50.961544 8011 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:28:50.963318 8011 webserver.cc:460] Webserver started at http://127.5.232.130:45079/ using document root <none> and password file <none>
I20260929 16:28:50.963992 8011 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:50.964099 8011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:50.964425 8011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:50.967425 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "499388d1f2754abc994531cbee83d442"
format_stamp: "Formatted at 2026-09-29 16:28:50 on dist-test-slave-dssx"
server_key: "332e56937e73bb46f2c2f9d6ed47ce84"
server_key_iv: "da929a12125410ac096c81875c5da36b"
server_key_version: "encryptionkey@0"
I20260929 16:28:50.968149 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "499388d1f2754abc994531cbee83d442"
format_stamp: "Formatted at 2026-09-29 16:28:50 on dist-test-slave-dssx"
server_key: "332e56937e73bb46f2c2f9d6ed47ce84"
server_key_iv: "da929a12125410ac096c81875c5da36b"
server_key_version: "encryptionkey@0"
I20260929 16:28:50.973625 8011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:28:50.977264 8026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:50.978883 8011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:28:50.979069 8011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "499388d1f2754abc994531cbee83d442"
format_stamp: "Formatted at 2026-09-29 16:28:50 on dist-test-slave-dssx"
server_key: "332e56937e73bb46f2c2f9d6ed47ce84"
server_key_iv: "da929a12125410ac096c81875c5da36b"
server_key_version: "encryptionkey@0"
I20260929 16:28:50.979322 8011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:51.003371 8011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:51.007551 8011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:51.007869 8011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:51.010800 8011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:51.010938 8011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:51.011072 8011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:51.011157 8011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:28:51 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699330, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:51.036702 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.011252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:48919 (local address 127.5.232.190:36861)
0929 16:28:51.011452 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.011460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.013134 (+ 1674us) server_negotiation.cc:407] Connection header received
0929 16:28:51.016336 (+ 3202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.016343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.016453 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.016535 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:51.019577 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.020676 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.023209 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.023859 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.024003 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.029127 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:51.029136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:51.029141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:51.029187 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:51.032634 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.033720 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.033727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.033730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.033852 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.034515 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.034521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.034524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.034934 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:51.035095 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.036130 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.036350 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:28:51.038187 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.011999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36861 (local address 127.5.232.130:48919)
0929 16:28:51.012940 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.012998 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.015863 (+ 2865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.017244 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.017257 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.018214 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:51.019366 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.019400 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.020900 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.020923 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.023029 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.023050 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.024126 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.024134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.024491 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.025356 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:51.025435 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:51.028776 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:51.032792 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.032805 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.032826 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.033555 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.034055 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.034062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.034068 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.034387 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.035166 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:51.035172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:51.035743 (+ 571us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.036041 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.036652 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":376,"threads_started":1}
I20260929 16:28:51.059157 8011 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:42019
I20260929 16:28:51.059180 8145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:42019 every 8 connection(s)
I20260929 16:28:51.060549 8011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:28:51.061014 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.045901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:41483 (local address 127.5.232.190:36861)
0929 16:28:51.046146 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.046156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.046811 (+ 655us) server_negotiation.cc:407] Connection header received
0929 16:28:51.047087 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.047092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.047183 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.047306 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:51.048939 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.049709 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.051810 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.052406 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.052547 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.054871 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:51.054879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:51.054885 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:51.054933 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:51.058148 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.058972 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.058977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.058981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.059118 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.059572 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.059577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.059581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.059917 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:51.060070 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.060443 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.060667 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:28:51.061172 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.045891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36861 (local address 127.5.232.130:41483)
0929 16:28:51.046681 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.046708 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.046934 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.047336 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.047340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.048151 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:51.048781 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.048795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.049886 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.049891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.051657 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.051673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.052594 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.052600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.052752 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.053503 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:51.053518 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:51.054667 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:51.058344 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.058349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.058353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.058806 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.059234 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.059239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.059243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.059460 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.060032 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:51.060037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:51.060213 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.060354 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.060691 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":368,"threads_started":1}
I20260929 16:28:51.062960 8146 heartbeater.cc:347] Connected to a master server at 127.5.232.190:36861
I20260929 16:28:51.063206 8146 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:51.063933 8146 heartbeater.cc:511] Master 127.5.232.190:36861 requested a full tablet report, sending...
I20260929 16:28:51.065261 7800 ts_manager.cc:194] Registered new tserver with Master: 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019)
I20260929 16:28:51.065304 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 8011
I20260929 16:28:51.065490 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:28:51.065943 6050 external_mini_cluster.cc:1428] Setting key 19047cb95459916cd8e8d3fcc76de4ae
I20260929 16:28:51.066200 7800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:48919
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:28:51.106335 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:36861
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:51.439833 8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:51.440194 8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:51.440366 8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:51.448015 8150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:51.448205 8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:51.448354 8150 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:28:51.448441 8150 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:28:51.448578 8150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:28:51.459470 8150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.232.190:36861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:51.460893 8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:51.462373 8150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:51.476311 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:51.476251 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:51.476313 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:51.477255 8150 server_base.cc:1034] running on GCE node
I20260929 16:28:51.477985 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:51.478955 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:51.480187 8150 hybrid_clock.cc:648] HybridClock initialized: now 1790699331480146 us; error 57 us; skew 500 ppm
I20260929 16:28:51.480515 8150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:51 dist-test-slave-dssx krb5kdc[7755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:51.483831 8150 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:28:51.485567 8150 webserver.cc:460] Webserver started at http://127.5.232.131:34411/ using document root <none> and password file <none>
I20260929 16:28:51.486243 8150 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:51.486344 8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:51.486608 8150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:51.489542 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "279a707487664029b87b0ffbc5411148"
format_stamp: "Formatted at 2026-09-29 16:28:51 on dist-test-slave-dssx"
server_key: "167582fc44293b7e826cc142e2bfb3b3"
server_key_iv: "860ebed295029f0b7052899ef312a80b"
server_key_version: "encryptionkey@0"
I20260929 16:28:51.490393 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "279a707487664029b87b0ffbc5411148"
format_stamp: "Formatted at 2026-09-29 16:28:51 on dist-test-slave-dssx"
server_key: "167582fc44293b7e826cc142e2bfb3b3"
server_key_iv: "860ebed295029f0b7052899ef312a80b"
server_key_version: "encryptionkey@0"
I20260929 16:28:51.495824 8150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:28:51.499634 8165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:51.501168 8150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:28:51.501317 8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "279a707487664029b87b0ffbc5411148"
format_stamp: "Formatted at 2026-09-29 16:28:51 on dist-test-slave-dssx"
server_key: "167582fc44293b7e826cc142e2bfb3b3"
server_key_iv: "860ebed295029f0b7052899ef312a80b"
server_key_version: "encryptionkey@0"
I20260929 16:28:51.501593 8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:51.524626 8150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:51.529222 8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:51.529560 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:51.532230 8150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:51.532346 8150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:51.532562 8150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:51.532672 8150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:28:51 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:51.552959 8007 heartbeater.cc:503] Master 127.5.232.190:36861 was elected leader, sending a full tablet report...
I20260929 16:28:51.556628 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.532813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:45003 (local address 127.5.232.190:36861)
0929 16:28:51.533069 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.533077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.534633 (+ 1556us) server_negotiation.cc:407] Connection header received
0929 16:28:51.536733 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.536739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.536831 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.536936 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:51.539319 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.540564 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.544408 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.544818 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.544965 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.549501 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:51.549507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:51.549511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:51.549544 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:51.552179 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.553332 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.553340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.553345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.553560 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.554153 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.554157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.554159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.554438 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:51.554606 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.555705 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.556080 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:28:51.557207 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.533531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36861 (local address 127.5.232.131:45003)
0929 16:28:51.534433 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.534489 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.536341 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.537570 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.537585 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.538312 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:51.539068 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.539100 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.540831 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.540839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.543674 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.543691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.545078 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.545086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.545506 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.546473 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:51.546500 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:51.549144 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:51.552342 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.552351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.552370 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.553197 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.553669 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.553675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.553678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.553991 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.554674 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:51.554680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:51.555201 (+ 521us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.555479 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.555883 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":391,"threads_started":1}
I20260929 16:28:51.580113 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.564555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:43045 (local address 127.5.232.190:36861)
0929 16:28:51.564820 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.564829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.565500 (+ 671us) server_negotiation.cc:407] Connection header received
0929 16:28:51.565738 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.565743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.565836 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.565953 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:51.567502 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.568416 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.570620 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.571230 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.571397 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.574090 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:51.574097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:51.574102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:51.574189 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:51.576790 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.577961 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.577967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.577971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.578155 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.578599 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.578605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.578608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.578900 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:51.579060 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.579504 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.579737 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:28:51.580639 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.564658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36861 (local address 127.5.232.131:43045)
0929 16:28:51.565330 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.565349 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.565579 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.566018 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.566026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.566846 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:51.567333 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.567352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.568634 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.568644 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.570445 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.570461 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.571448 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.571459 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.571650 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.572452 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:51.572473 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:51.573903 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:51.576972 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.576980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.576988 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.577741 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.578290 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.578297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.578306 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.578469 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.579025 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:51.579030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:51.579226 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.579395 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.579754 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":375,"threads_started":1}
I20260929 16:28:51.588797 8150 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:43465
I20260929 16:28:51.588847 8284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:43465 every 8 connection(s)
I20260929 16:28:51.590170 8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:28:51.592244 8285 heartbeater.cc:347] Connected to a master server at 127.5.232.190:36861
I20260929 16:28:51.592495 8285 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:51.593146 8285 heartbeater.cc:511] Master 127.5.232.190:36861 requested a full tablet report, sending...
I20260929 16:28:51.594596 7800 ts_manager.cc:194] Registered new tserver with Master: 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465)
I20260929 16:28:51.595388 7800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:45003
I20260929 16:28:51.595434 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 8150
I20260929 16:28:51.595603 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:28:51.596135 6050 external_mini_cluster.cc:1428] Setting key 3c5fa8d66e031154a846eb68c8959999
I20260929 16:28:51.599002 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:28:51 dist-test-slave-dssx krb5kdc[7755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:28:51 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:51.646427 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.632214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.5.232.190:36861)
0929 16:28:51.632465 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.632472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.632673 (+ 201us) server_negotiation.cc:407] Connection header received
0929 16:28:51.632917 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.632922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.632987 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.633096 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:51.634561 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.635291 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.636589 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.637032 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.637184 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.641116 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:51.641121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:51.641125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:51.641162 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:51.643695 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.644384 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.644394 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.644397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.644527 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.644914 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.644920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.644924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.645206 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:51.645506 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.645807 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.646050 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 16:28:51.650564 7800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:28:51.657613 7800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:51.685738 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.670589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.5.232.129:46825)
0929 16:28:51.673413 (+ 2824us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.673423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.674441 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 16:28:51.674592 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.674605 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.674938 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.675176 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:51.676583 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.678779 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.682070 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.683619 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.683762 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.684008 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.685178 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.685367 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2644,"thread_start_us":2515,"threads_started":1}
I20260929 16:28:51.685760 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.670188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.0.0.1:50514)
0929 16:28:51.672943 (+ 2755us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.673552 (+ 609us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.674186 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.675520 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.675531 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.675574 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:51.676263 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.676307 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.678992 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.679002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.681890 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.681903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.684163 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.684173 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.684886 (+ 713us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.684950 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.685116 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.685322 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2406,"thread_start_us":9171,"threads_started":1}
I20260929 16:28:51.688879 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.673275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.5.232.130:42019)
0929 16:28:51.680953 (+ 7678us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.680961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.680991 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:28:51.681087 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.681095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.681314 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.681572 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:51.682381 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.684048 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.686790 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.687827 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.687993 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.688211 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.688429 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.688590 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7512,"thread_start_us":7376,"threads_started":1}
I20260929 16:28:51.688951 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.672780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:42019 (local address 127.0.0.1:52716)
0929 16:28:51.680213 (+ 7433us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.680238 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.680407 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.681573 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.681577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.681601 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:51.682171 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.682188 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.684197 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.684201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.686617 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.686631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.688044 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.688051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.688215 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.688253 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.688375 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.688507 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7258,"mutex_wait_us":31,"thread_start_us":7060,"threads_started":1}
I20260929 16:28:51.690616 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.670461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:43465 (local address 127.0.0.1:53672)
0929 16:28:51.679654 (+ 9193us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.679677 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.680856 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.681863 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.681871 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.681901 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:51.682410 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.682418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.684188 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.684193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.686542 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.686554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.689839 (+ 3285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.689845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.690012 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.690042 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.690140 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.690263 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9041,"mutex_wait_us":36,"thread_start_us":1941,"threads_started":1}
I20260929 16:28:51.690734 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.670755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.5.232.131:43465)
0929 16:28:51.672179 (+ 1424us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.672193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.679803 (+ 7610us) server_negotiation.cc:407] Connection header received
0929 16:28:51.681066 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.681077 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.681365 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.681557 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:51.682610 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.684025 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.686707 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.688389 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.688515 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.688728 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.690221 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.690408 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":972,"threads_started":1}
I20260929 16:28:51.690340 7937 tablet_service.cc:1467] Processing CreateTablet for tablet 5025765abd6e4dcfbb37fe3b24c0f66e (DEFAULT_TABLE table=test-table [id=f9123dccbc034032a73b9d112e0aabca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:51.691907 7937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025765abd6e4dcfbb37fe3b24c0f66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:51.692610 8076 tablet_service.cc:1467] Processing CreateTablet for tablet 5025765abd6e4dcfbb37fe3b24c0f66e (DEFAULT_TABLE table=test-table [id=f9123dccbc034032a73b9d112e0aabca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:51.693861 8076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025765abd6e4dcfbb37fe3b24c0f66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:51.694555 8215 tablet_service.cc:1467] Processing CreateTablet for tablet 5025765abd6e4dcfbb37fe3b24c0f66e (DEFAULT_TABLE table=test-table [id=f9123dccbc034032a73b9d112e0aabca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:51.695907 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5025765abd6e4dcfbb37fe3b24c0f66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:51.705801 8304 tablet_bootstrap.cc:492] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: Bootstrap starting.
I20260929 16:28:51.709647 8304 tablet_bootstrap.cc:654] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:51.710144 8305 tablet_bootstrap.cc:492] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: Bootstrap starting.
I20260929 16:28:51.711236 8304 log.cc:824] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:51.712674 8306 tablet_bootstrap.cc:492] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: Bootstrap starting.
I20260929 16:28:51.713979 8305 tablet_bootstrap.cc:654] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:51.714555 8304 tablet_bootstrap.cc:492] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: No bootstrap required, opened a new log
I20260929 16:28:51.714845 8304 ts_tablet_manager.cc:1397] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 16:28:51.715320 8305 log.cc:824] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:51.716331 8306 tablet_bootstrap.cc:654] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:51.717665 8306 log.cc:824] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:51.718670 8305 tablet_bootstrap.cc:492] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: No bootstrap required, opened a new log
I20260929 16:28:51.718976 8305 ts_tablet_manager.cc:1397] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:28:51.720613 8306 tablet_bootstrap.cc:492] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: No bootstrap required, opened a new log
I20260929 16:28:51.720911 8306 ts_tablet_manager.cc:1397] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 16:28:51.727242 8304 raft_consensus.cc:348] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } }
I20260929 16:28:51.727773 8304 raft_consensus.cc:374] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:51.727866 8304 raft_consensus.cc:729] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499388d1f2754abc994531cbee83d442, State: Initialized, Role: FOLLOWER
I20260929 16:28:51.728654 8304 consensus_queue.cc:260] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } }
I20260929 16:28:51.729756 8305 raft_consensus.cc:348] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } }
I20260929 16:28:51.730355 8305 raft_consensus.cc:374] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:51.730473 8305 raft_consensus.cc:729] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2243e6db114eda9fb2fbe25c468b17, State: Initialized, Role: FOLLOWER
I20260929 16:28:51.731436 8305 consensus_queue.cc:260] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } }
I20260929 16:28:51.732501 8306 raft_consensus.cc:348] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } }
I20260929 16:28:51.732959 8306 raft_consensus.cc:374] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:51.733064 8306 raft_consensus.cc:729] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279a707487664029b87b0ffbc5411148, State: Initialized, Role: FOLLOWER
I20260929 16:28:51.733757 8306 consensus_queue.cc:260] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } }
I20260929 16:28:51.734931 8285 heartbeater.cc:503] Master 127.5.232.190:36861 was elected leader, sending a full tablet report...
I20260929 16:28:51.735705 8306 ts_tablet_manager.cc:1428] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 16:28:51.736785 8146 heartbeater.cc:503] Master 127.5.232.190:36861 was elected leader, sending a full tablet report...
I20260929 16:28:51.737254 8305 ts_tablet_manager.cc:1428] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: Time spent starting tablet: real 0.018s user 0.009s sys 0.009s
I20260929 16:28:51.739198 8304 ts_tablet_manager.cc:1428] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: Time spent starting tablet: real 0.024s user 0.016s sys 0.005s
W20260929 16:28:51.793972 8008 tablet.cc:2369] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:28:51.813007 8147 tablet.cc:2369] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:28:51.828250 8312 raft_consensus.cc:482] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:51.828624 8312 raft_consensus.cc:504] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } }
I20260929 16:28:51.830698 8312 leader_election.cc:288] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825), 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019)
I20260929 16:28:51.837306 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.831199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:39155 (local address 127.5.232.129:46825)
0929 16:28:51.831415 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.831421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.831548 (+ 127us) server_negotiation.cc:407] Connection header received
0929 16:28:51.831783 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.831788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.831849 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.831952 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:51.832402 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.833431 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.835377 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.836477 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.836602 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.836654 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.836995 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.837088 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:28:51.837646 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.831141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.5.232.131:39155)
0929 16:28:51.831438 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.831464 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.831614 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.831970 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.831975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.832001 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:51.832243 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.832257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.833572 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.833578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.835212 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.835224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.836624 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.836631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.836818 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.836856 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.836953 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.837099 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":3}
I20260929 16:28:51.838181 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.831264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:53215 (local address 127.5.232.130:42019)
0929 16:28:51.831507 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.831516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.832160 (+ 644us) server_negotiation.cc:407] Connection header received
0929 16:28:51.832302 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.832310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.832393 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.832470 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:51.833043 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.834159 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.836158 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.837185 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.837340 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.837441 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.837776 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.837905 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 16:28:51.838205 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.831264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:42019 (local address 127.5.232.131:53215)
0929 16:28:51.832050 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.832064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.832178 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.832532 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.832536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.832560 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:51.832891 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.832903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.834307 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.834312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.835981 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.835998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.837451 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.837459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.837619 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.837642 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.837751 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.837890 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":452,"threads_started":1}
I20260929 16:28:51.838867 7957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025765abd6e4dcfbb37fe3b24c0f66e" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2243e6db114eda9fb2fbe25c468b17" is_pre_election: true
I20260929 16:28:51.839190 8096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025765abd6e4dcfbb37fe3b24c0f66e" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442" is_pre_election: true
I20260929 16:28:51.839462 7957 raft_consensus.cc:2454] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 0.
I20260929 16:28:51.839740 8096 raft_consensus.cc:2454] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 0.
I20260929 16:28:51.840394 8169 leader_election.cc:302] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 279a707487664029b87b0ffbc5411148, 499388d1f2754abc994531cbee83d442, 9d2243e6db114eda9fb2fbe25c468b17; no voters:
I20260929 16:28:51.840869 8312 raft_consensus.cc:2781] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:51.841032 8312 raft_consensus.cc:482] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:51.841110 8312 raft_consensus.cc:3037] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:28:51.841671 8286 tablet.cc:2369] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:28:51.843803 8312 raft_consensus.cc:504] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } }
I20260929 16:28:51.844401 8312 leader_election.cc:288] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 election: Requested vote from peers 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825), 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019)
I20260929 16:28:51.844892 7957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025765abd6e4dcfbb37fe3b24c0f66e" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2243e6db114eda9fb2fbe25c468b17"
I20260929 16:28:51.844929 8096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5025765abd6e4dcfbb37fe3b24c0f66e" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442"
I20260929 16:28:51.845103 7957 raft_consensus.cc:3037] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:51.845180 8096 raft_consensus.cc:3037] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:51.847419 7957 raft_consensus.cc:2454] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 1.
I20260929 16:28:51.847429 8096 raft_consensus.cc:2454] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 1.
I20260929 16:28:51.847915 8168 leader_election.cc:302] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279a707487664029b87b0ffbc5411148, 499388d1f2754abc994531cbee83d442; no voters:
I20260929 16:28:51.848269 8312 raft_consensus.cc:2781] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:51.848724 8312 raft_consensus.cc:686] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 1 LEADER]: Becoming Leader. State: Replica: 279a707487664029b87b0ffbc5411148, State: Running, Role: LEADER
I20260929 16:28:51.849493 8312 consensus_queue.cc:237] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } }
I20260929 16:28:51.853987 7800 catalog_manager.cc:5495] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 reported cstate change: term changed from 0 to 1, leader changed from <none> to 279a707487664029b87b0ffbc5411148 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "279a707487664029b87b0ffbc5411148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:28:51 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699329, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:51.930732 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.912402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.5.232.190:36861)
0929 16:28:51.912634 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.912642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.913229 (+ 587us) server_negotiation.cc:407] Connection header received
0929 16:28:51.913511 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.913516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.913601 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.913704 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:51.915273 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.916511 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.918624 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.919421 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.919557 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.924305 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:51.924312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:51.924316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:51.924366 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:51.927378 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.928257 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.928264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.928269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.928466 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:51.928845 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:51.928850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:51.928854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:51.929282 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:51.929528 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.930071 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.930316 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:28:51.930895 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.912455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36861 (local address 127.0.0.1:60350)
0929 16:28:51.913107 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.913128 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.913311 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.913745 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.913749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.914768 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:51.915129 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.915142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.916652 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.916659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.918424 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.918443 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.919606 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.919611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.919790 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.920604 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:51.920637 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:51.924085 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:51.927532 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.927548 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.927558 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.928124 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.928580 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:51.928584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:51.928588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:51.928732 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:51.929533 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:51.929544 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:51.929759 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.929974 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.930456 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":116,"thread_start_us":361,"threads_started":1}
I20260929 16:28:51.936868 7800 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:60350:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:28:51.938170 7800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:51.945303 7937 tablet_service.cc:1467] Processing CreateTablet for tablet cccf600e39f24326ab7c66a3e5f7ccd0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37eb9258d3df48feb7f26dd40001ef21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:28:51.945706 8076 tablet_service.cc:1467] Processing CreateTablet for tablet cccf600e39f24326ab7c66a3e5f7ccd0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37eb9258d3df48feb7f26dd40001ef21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:28:51.945760 8215 tablet_service.cc:1467] Processing CreateTablet for tablet cccf600e39f24326ab7c66a3e5f7ccd0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37eb9258d3df48feb7f26dd40001ef21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:28:51.946410 7937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf600e39f24326ab7c66a3e5f7ccd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:51.946467 8076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf600e39f24326ab7c66a3e5f7ccd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:51.946538 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf600e39f24326ab7c66a3e5f7ccd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:51.951963 8304 tablet_bootstrap.cc:492] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: Bootstrap starting.
I20260929 16:28:51.953769 8305 tablet_bootstrap.cc:492] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: Bootstrap starting.
I20260929 16:28:51.953800 8304 tablet_bootstrap.cc:654] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:51.955293 8305 tablet_bootstrap.cc:654] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:51.956305 8304 tablet_bootstrap.cc:492] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: No bootstrap required, opened a new log
I20260929 16:28:51.956475 8304 ts_tablet_manager.cc:1397] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:28:51.956601 8306 tablet_bootstrap.cc:492] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: Bootstrap starting.
I20260929 16:28:51.957267 8304 raft_consensus.cc:348] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:51.957588 8304 raft_consensus.cc:374] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:51.957693 8304 raft_consensus.cc:729] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499388d1f2754abc994531cbee83d442, State: Initialized, Role: FOLLOWER
I20260929 16:28:51.957733 8305 tablet_bootstrap.cc:492] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: No bootstrap required, opened a new log
I20260929 16:28:51.957868 8305 ts_tablet_manager.cc:1397] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:28:51.958003 8304 consensus_queue.cc:260] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:51.958369 8306 tablet_bootstrap.cc:654] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:51.958480 8310 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:51.958875 8310 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:51.958684 8305 raft_consensus.cc:348] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:51.958992 8305 raft_consensus.cc:374] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:51.959089 8305 raft_consensus.cc:729] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2243e6db114eda9fb2fbe25c468b17, State: Initialized, Role: FOLLOWER
I20260929 16:28:51.959954 8304 ts_tablet_manager.cc:1428] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:51.959743 8305 consensus_queue.cc:260] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:51.960309 8311 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:51.960642 8311 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:51.961642 8305 ts_tablet_manager.cc:1428] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:28:51.962869 8306 tablet_bootstrap.cc:492] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: No bootstrap required, opened a new log
I20260929 16:28:51.963048 8306 ts_tablet_manager.cc:1397] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260929 16:28:51.963893 8306 raft_consensus.cc:348] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:51.964208 8306 raft_consensus.cc:374] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:51.964318 8306 raft_consensus.cc:729] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279a707487664029b87b0ffbc5411148, State: Initialized, Role: FOLLOWER
I20260929 16:28:51.964577 8306 consensus_queue.cc:260] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:51.964991 8312 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:51.965179 8306 ts_tablet_manager.cc:1428] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:28:51.965318 8312 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:51.982931 8311 raft_consensus.cc:482] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:51.983227 8311 raft_consensus.cc:504] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:51.985010 8311 leader_election.cc:288] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019), 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465)
I20260929 16:28:51.992031 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.985479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:43121 (local address 127.5.232.131:43465)
0929 16:28:51.985731 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.985738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.986467 (+ 729us) server_negotiation.cc:407] Connection header received
0929 16:28:51.986631 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.986637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.986712 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.986817 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:51.987379 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.988076 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.990180 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.990998 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.991198 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.991294 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.991613 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.991750 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:28:51.992110 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.985904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:60577 (local address 127.5.232.130:42019)
0929 16:28:51.986082 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:51.986090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:51.986110 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:28:51.986395 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:51.986404 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:51.986489 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:51.986587 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:51.987283 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.988097 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.990181 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.990998 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.991207 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.991291 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:51.991675 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 16:28:51.991810 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:28:51.992210 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.985485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:43465 (local address 127.5.232.129:43121)
0929 16:28:51.986353 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.986371 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.986505 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.986842 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.986845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.986863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:51.987226 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.987237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.988229 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.988235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.989990 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.990005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.991261 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.991268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.991441 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.991466 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.991547 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.991710 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":82,"thread_start_us":628,"threads_started":1}
I20260929 16:28:51.992269 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.985250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:42019 (local address 127.5.232.129:60577)
0929 16:28:51.986021 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.986041 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:51.986198 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:51.986627 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:51.986632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:51.986659 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:51.987077 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.987089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.988269 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.988272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.989998 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:51.990011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:51.991325 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:51.991334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:51.991482 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:51.991505 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:28:51.991619 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:28:51.991750 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":11,"thread_start_us":448,"threads_started":1}
I20260929 16:28:51.993112 8096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf600e39f24326ab7c66a3e5f7ccd0" candidate_uuid: "9d2243e6db114eda9fb2fbe25c468b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442" is_pre_election: true
I20260929 16:28:51.993129 8240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf600e39f24326ab7c66a3e5f7ccd0" candidate_uuid: "9d2243e6db114eda9fb2fbe25c468b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279a707487664029b87b0ffbc5411148" is_pre_election: true
I20260929 16:28:51.993374 8096 raft_consensus.cc:2454] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2243e6db114eda9fb2fbe25c468b17 in term 0.
I20260929 16:28:51.993644 8240 raft_consensus.cc:2454] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2243e6db114eda9fb2fbe25c468b17 in term 0.
I20260929 16:28:51.993961 7889 leader_election.cc:302] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499388d1f2754abc994531cbee83d442, 9d2243e6db114eda9fb2fbe25c468b17; no voters:
I20260929 16:28:51.994468 8311 raft_consensus.cc:2781] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:51.994630 8311 raft_consensus.cc:482] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:51.994729 8311 raft_consensus.cc:3037] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:51.996812 8311 raft_consensus.cc:504] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:51.997471 8311 leader_election.cc:288] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 election: Requested vote from peers 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019), 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465)
I20260929 16:28:51.997952 8096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf600e39f24326ab7c66a3e5f7ccd0" candidate_uuid: "9d2243e6db114eda9fb2fbe25c468b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442"
I20260929 16:28:51.997960 8240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf600e39f24326ab7c66a3e5f7ccd0" candidate_uuid: "9d2243e6db114eda9fb2fbe25c468b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279a707487664029b87b0ffbc5411148"
I20260929 16:28:51.998158 8240 raft_consensus.cc:3037] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:51.998165 8096 raft_consensus.cc:3037] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:52.000212 8096 raft_consensus.cc:2454] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2243e6db114eda9fb2fbe25c468b17 in term 1.
I20260929 16:28:52.000450 8240 raft_consensus.cc:2454] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2243e6db114eda9fb2fbe25c468b17 in term 1.
I20260929 16:28:52.000680 7889 leader_election.cc:302] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499388d1f2754abc994531cbee83d442, 9d2243e6db114eda9fb2fbe25c468b17; no voters:
I20260929 16:28:52.001044 8311 raft_consensus.cc:2781] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:52.001494 8311 raft_consensus.cc:686] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 LEADER]: Becoming Leader. State: Replica: 9d2243e6db114eda9fb2fbe25c468b17, State: Running, Role: LEADER
I20260929 16:28:52.002275 8311 consensus_queue.cc:237] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.005664 8327 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: TxnStatusTablet state changed. Reason: New leader 9d2243e6db114eda9fb2fbe25c468b17. Latest consensus state: current_term: 1 leader_uuid: "9d2243e6db114eda9fb2fbe25c468b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:52.006100 8327 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:28:52.006865 8329 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:28:52.007205 8329 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:28:52.007725 7799 catalog_manager.cc:5495] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129). New cstate: current_term: 1 leader_uuid: "9d2243e6db114eda9fb2fbe25c468b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } health_report { overall_health: UNKNOWN } } }
I20260929 16:28:52.033471 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.026978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.0.0.1:50528)
0929 16:28:52.027197 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:52.027220 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:52.027359 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:52.027860 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:52.027863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:52.027873 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:28:52.028026 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:52.028033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.029583 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.029587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:52.030712 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:52.030727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.031680 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.031685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:52.031834 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:52.031898 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:28:52.032970 (+ 1072us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:28:52.032986 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 16:28:52.033040 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 16:28:52.033164 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:28:52.033546 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.026977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.5.232.129:46825)
0929 16:28:52.027181 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:52.027187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:52.027310 (+ 123us) server_negotiation.cc:407] Connection header received
0929 16:28:52.027563 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:52.027569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:52.027678 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:52.027841 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:52.028150 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.029446 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.030855 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.031522 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.031658 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:52.032101 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:52.032852 (+ 751us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:52.033019 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:52.033070 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:28:52.033160 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 16:28:52.041640 8240 raft_consensus.cc:1261] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Refusing update from remote peer 9d2243e6db114eda9fb2fbe25c468b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:52.041746 8096 raft_consensus.cc:1261] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Refusing update from remote peer 9d2243e6db114eda9fb2fbe25c468b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:52.042697 8327 consensus_queue.cc:1035] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:52.043063 8311 consensus_queue.cc:1035] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:52.058882 8310 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: TxnStatusTablet state changed. Reason: New leader 9d2243e6db114eda9fb2fbe25c468b17. Latest consensus state: current_term: 1 leader_uuid: "9d2243e6db114eda9fb2fbe25c468b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:52.059201 8310 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:52.059305 8312 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: TxnStatusTablet state changed. Reason: New leader 9d2243e6db114eda9fb2fbe25c468b17. Latest consensus state: current_term: 1 leader_uuid: "9d2243e6db114eda9fb2fbe25c468b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:52.059613 8312 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:52.073889 8312 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d2243e6db114eda9fb2fbe25c468b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:52.074216 8312 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 279a707487664029b87b0ffbc5411148: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:52.078181 8310 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d2243e6db114eda9fb2fbe25c468b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:52.078478 8310 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 499388d1f2754abc994531cbee83d442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:28:52.079526 8311 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d2243e6db114eda9fb2fbe25c468b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:52.079809 8311 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:28:52.080214 8311 tablet_replica.cc:439] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d2243e6db114eda9fb2fbe25c468b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } } }
I20260929 16:28:52.080447 8311 tablet_replica.cc:442] T cccf600e39f24326ab7c66a3e5f7ccd0 P 9d2243e6db114eda9fb2fbe25c468b17: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:28:52.092726 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.087119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.5.232.131:43465)
0929 16:28:52.087298 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:52.087304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:52.087511 (+ 207us) server_negotiation.cc:407] Connection header received
0929 16:28:52.087765 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:52.087769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:52.087851 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:52.088021 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:52.088376 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.089089 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.090516 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.090902 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.091052 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:52.091539 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:52.092089 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:52.092251 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:52.092332 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 16:28:52.092414 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:28:52.108531 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.102837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:43619 (local address 127.5.232.129:46825)
0929 16:28:52.103038 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:52.103048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:52.103538 (+ 490us) server_negotiation.cc:407] Connection header received
0929 16:28:52.103758 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:52.103763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:52.103836 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:52.103938 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:52.104400 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.105260 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.106750 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.107180 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.107332 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:52.107774 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:52.107871 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:52.107977 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:52.108124 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 16:28:52.108207 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:28:52.108632 8343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.102816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.5.232.131:43619)
0929 16:28:52.103422 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:52.103444 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:52.103614 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:52.103958 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:52.103962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:52.103973 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:28:52.104249 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:52.104256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.105418 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.105425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:52.106586 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:52.106598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.107365 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.107373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:52.107508 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:52.107567 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:28:52.107991 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:28:52.108004 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 16:28:52.108099 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:28:52.108236 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":315,"threads_started":1}
I20260929 16:28:52.118940 8096 raft_consensus.cc:1261] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Refusing update from remote peer 279a707487664029b87b0ffbc5411148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:52.119234 7957 raft_consensus.cc:1261] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Refusing update from remote peer 279a707487664029b87b0ffbc5411148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:52.119663 8314 consensus_queue.cc:1035] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:52.120378 8312 consensus_queue.cc:1035] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:52.134243 8335 mvcc.cc:205] Tried to move back new op lower bound from 7334704464352088064 to 7334704463259168768. Current Snapshot: MvccSnapshot[applied={T|T < 7334704464352088064}]
I20260929 16:28:52.140856 8332 mvcc.cc:205] Tried to move back new op lower bound from 7334704464352088064 to 7334704463259168768. Current Snapshot: MvccSnapshot[applied={T|T < 7334704464352088064}]
I20260929 16:28:52.171541 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.164417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:42891 (local address 127.5.232.131:43465)
0929 16:28:52.164651 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:52.164672 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:52.165360 (+ 688us) server_negotiation.cc:407] Connection header received
0929 16:28:52.165631 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:52.165637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:52.165741 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:52.165830 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:52.166479 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.167186 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.169378 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.170026 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.170158 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:52.170747 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:52.170873 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:52.170951 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:52.171241 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 16:28:52.171322 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:28:52.172405 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.164507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:43465 (local address 127.5.232.129:42891)
0929 16:28:52.165245 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:52.165264 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:52.165490 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:52.165859 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:52.165864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:52.165879 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:28:52.166332 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:52.166344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.167316 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.167322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:52.169218 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:52.169234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:52.170224 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:52.170230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:52.170454 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:52.170541 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:28:52.171098 (+ 557us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:28:52.171107 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:28:52.171976 (+ 869us) client_negotiation.cc:240] Negotiation successful
0929 16:28:52.172109 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":371,"threads_started":1}
I20260929 16:28:52.210026 7800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:52.210328 7800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:52.214540 7800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018: Sending DeleteTablet for 3 replicas of tablet 5025765abd6e4dcfbb37fe3b24c0f66e
I20260929 16:28:52.215821 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 5025765abd6e4dcfbb37fe3b24c0f66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:52 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:53672
I20260929 16:28:52.215822 7937 tablet_service.cc:1514] Processing DeleteTablet for tablet 5025765abd6e4dcfbb37fe3b24c0f66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:52 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:50514
I20260929 16:28:52.217662 8076 tablet_service.cc:1514] Processing DeleteTablet for tablet 5025765abd6e4dcfbb37fe3b24c0f66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:52 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:52716
I20260929 16:28:52.225136 7800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:28:52.227221 7800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:52.227591 8353 tablet_replica.cc:330] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: stopping tablet replica
I20260929 16:28:52.228148 8353 raft_consensus.cc:2229] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:52.228575 8353 raft_consensus.cc:2258] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:52.230618 8353 ts_tablet_manager.cc:1905] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:52.239692 7937 tablet_service.cc:1467] Processing CreateTablet for tablet 2d37b9a5646844a3b77faf86860ae90f (DEFAULT_TABLE table=test-table [id=0e7b752486ac40dcaed249674ba6ff7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:52.240129 7937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d37b9a5646844a3b77faf86860ae90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:52.240216 8214 tablet_service.cc:1467] Processing CreateTablet for tablet 2d37b9a5646844a3b77faf86860ae90f (DEFAULT_TABLE table=test-table [id=0e7b752486ac40dcaed249674ba6ff7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:52.240592 8075 tablet_service.cc:1467] Processing CreateTablet for tablet 2d37b9a5646844a3b77faf86860ae90f (DEFAULT_TABLE table=test-table [id=0e7b752486ac40dcaed249674ba6ff7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:52.240778 8214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d37b9a5646844a3b77faf86860ae90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:52.241086 8075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d37b9a5646844a3b77faf86860ae90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:52.246780 8305 tablet_bootstrap.cc:492] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: Bootstrap starting.
I20260929 16:28:52.247501 8306 tablet_bootstrap.cc:492] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: Bootstrap starting.
I20260929 16:28:52.247675 8304 tablet_bootstrap.cc:492] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: Bootstrap starting.
I20260929 16:28:52.248512 8305 tablet_bootstrap.cc:654] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:52.249163 8306 tablet_bootstrap.cc:654] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:52.249261 8304 tablet_bootstrap.cc:654] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:52.239274 8353 ts_tablet_manager.cc:1918] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:28:52.256989 8353 log.cc:1196] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/5025765abd6e4dcfbb37fe3b24c0f66e
I20260929 16:28:52.259536 8353 ts_tablet_manager.cc:1939] T 5025765abd6e4dcfbb37fe3b24c0f66e P 9d2243e6db114eda9fb2fbe25c468b17: Deleting consensus metadata
I20260929 16:28:52.262876 8355 tablet_replica.cc:330] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: stopping tablet replica
I20260929 16:28:52.263491 8355 raft_consensus.cc:2229] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:52.263983 8355 raft_consensus.cc:2258] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:52.265012 8354 tablet_replica.cc:330] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: stopping tablet replica
I20260929 16:28:52.265020 7787 catalog_manager.cc:4841] TS 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825): tablet 5025765abd6e4dcfbb37fe3b24c0f66e (table test-table [id=f9123dccbc034032a73b9d112e0aabca]) successfully deleted
I20260929 16:28:52.265456 8354 raft_consensus.cc:2229] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:28:52.266124 8354 raft_consensus.cc:2258] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:52.268200 8355 ts_tablet_manager.cc:1905] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:52.268553 8306 tablet_bootstrap.cc:492] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: No bootstrap required, opened a new log
I20260929 16:28:52.268731 8306 ts_tablet_manager.cc:1397] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.001s
I20260929 16:28:52.269630 8306 raft_consensus.cc:348] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.269886 8306 raft_consensus.cc:374] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:52.269954 8306 raft_consensus.cc:729] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279a707487664029b87b0ffbc5411148, State: Initialized, Role: FOLLOWER
I20260929 16:28:52.269810 8305 tablet_bootstrap.cc:492] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: No bootstrap required, opened a new log
I20260929 16:28:52.270234 8305 ts_tablet_manager.cc:1397] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.003s
I20260929 16:28:52.270363 8304 tablet_bootstrap.cc:492] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: No bootstrap required, opened a new log
I20260929 16:28:52.270236 8306 consensus_queue.cc:260] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.270815 8306 ts_tablet_manager.cc:1428] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:28:52.271617 8304 ts_tablet_manager.cc:1397] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.000s
I20260929 16:28:52.272374 8304 raft_consensus.cc:348] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.272524 8305 raft_consensus.cc:348] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.272960 8305 raft_consensus.cc:374] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:52.273094 8305 raft_consensus.cc:729] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2243e6db114eda9fb2fbe25c468b17, State: Initialized, Role: FOLLOWER
I20260929 16:28:52.273429 8305 consensus_queue.cc:260] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.273850 8304 raft_consensus.cc:374] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:52.273950 8304 raft_consensus.cc:729] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499388d1f2754abc994531cbee83d442, State: Initialized, Role: FOLLOWER
I20260929 16:28:52.274235 8304 consensus_queue.cc:260] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.274956 8305 ts_tablet_manager.cc:1428] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 16:28:52.275851 8304 ts_tablet_manager.cc:1428] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:28:52.277359 8354 ts_tablet_manager.cc:1905] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:52.285614 8354 ts_tablet_manager.cc:1918] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:28:52.285801 8354 log.cc:1196] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/5025765abd6e4dcfbb37fe3b24c0f66e
I20260929 16:28:52.286360 8354 ts_tablet_manager.cc:1939] T 5025765abd6e4dcfbb37fe3b24c0f66e P 279a707487664029b87b0ffbc5411148: Deleting consensus metadata
I20260929 16:28:52.287959 8355 ts_tablet_manager.cc:1918] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:28:52.288192 8355 log.cc:1196] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/5025765abd6e4dcfbb37fe3b24c0f66e
I20260929 16:28:52.288318 7785 catalog_manager.cc:4841] TS 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465): tablet 5025765abd6e4dcfbb37fe3b24c0f66e (table test-table [id=f9123dccbc034032a73b9d112e0aabca]) successfully deleted
I20260929 16:28:52.288829 8355 ts_tablet_manager.cc:1939] T 5025765abd6e4dcfbb37fe3b24c0f66e P 499388d1f2754abc994531cbee83d442: Deleting consensus metadata
I20260929 16:28:52.290665 7786 catalog_manager.cc:4841] TS 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019): tablet 5025765abd6e4dcfbb37fe3b24c0f66e (table test-table [id=f9123dccbc034032a73b9d112e0aabca]) successfully deleted
I20260929 16:28:52.359778 8348 raft_consensus.cc:482] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:52.360030 8348 raft_consensus.cc:504] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.360797 8348 leader_election.cc:288] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825), 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019)
I20260929 16:28:52.361598 7957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d37b9a5646844a3b77faf86860ae90f" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2243e6db114eda9fb2fbe25c468b17" is_pre_election: true
I20260929 16:28:52.361671 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d37b9a5646844a3b77faf86860ae90f" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442" is_pre_election: true
I20260929 16:28:52.361857 7957 raft_consensus.cc:2454] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 0.
I20260929 16:28:52.361949 8095 raft_consensus.cc:2454] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 0.
I20260929 16:28:52.362381 8169 leader_election.cc:302] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279a707487664029b87b0ffbc5411148, 9d2243e6db114eda9fb2fbe25c468b17; no voters:
I20260929 16:28:52.362673 8348 raft_consensus.cc:2781] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:52.362797 8348 raft_consensus.cc:482] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:52.362900 8348 raft_consensus.cc:3037] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:52.366189 8348 raft_consensus.cc:504] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.367355 7957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d37b9a5646844a3b77faf86860ae90f" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2243e6db114eda9fb2fbe25c468b17"
I20260929 16:28:52.367587 7957 raft_consensus.cc:3037] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:52.368290 8095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d37b9a5646844a3b77faf86860ae90f" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442"
I20260929 16:28:52.368495 8095 raft_consensus.cc:3037] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:52.371042 7957 raft_consensus.cc:2454] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 1.
I20260929 16:28:52.371515 8095 raft_consensus.cc:2454] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 1.
I20260929 16:28:52.371544 8169 leader_election.cc:302] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279a707487664029b87b0ffbc5411148, 9d2243e6db114eda9fb2fbe25c468b17; no voters:
I20260929 16:28:52.371874 8314 raft_consensus.cc:2781] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:52.372074 8314 raft_consensus.cc:686] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 1 LEADER]: Becoming Leader. State: Replica: 279a707487664029b87b0ffbc5411148, State: Running, Role: LEADER
I20260929 16:28:52.372289 8348 leader_election.cc:288] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 election: Requested vote from peers 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825), 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019)
I20260929 16:28:52.372366 8314 consensus_queue.cc:237] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:52.375633 7798 catalog_manager.cc:5495] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 reported cstate change: term changed from 0 to 1, leader changed from <none> to 279a707487664029b87b0ffbc5411148 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "279a707487664029b87b0ffbc5411148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } health_report { overall_health: HEALTHY } } }
I20260929 16:28:52.407305 8095 raft_consensus.cc:1261] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Refusing update from remote peer 279a707487664029b87b0ffbc5411148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:52.407402 7957 raft_consensus.cc:1261] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Refusing update from remote peer 279a707487664029b87b0ffbc5411148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:52.408208 8312 consensus_queue.cc:1035] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:52.408658 8314 consensus_queue.cc:1035] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:28:52.447311 8335 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:28:54.041685 7798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:54.041953 7798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:54.045921 7798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018: Sending DeleteTablet for 3 replicas of tablet 2d37b9a5646844a3b77faf86860ae90f
I20260929 16:28:54.046749 7937 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d37b9a5646844a3b77faf86860ae90f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:54 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:50514
I20260929 16:28:54.046765 8076 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d37b9a5646844a3b77faf86860ae90f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:54 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:52716
I20260929 16:28:54.046983 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d37b9a5646844a3b77faf86860ae90f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:54 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:53672
I20260929 16:28:54.058245 8387 tablet_replica.cc:330] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: stopping tablet replica
I20260929 16:28:54.058647 8387 raft_consensus.cc:2229] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:28:54.059175 8387 raft_consensus.cc:2258] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:54.060345 8387 ts_tablet_manager.cc:1905] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:54.060830 8385 tablet_replica.cc:330] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: stopping tablet replica
I20260929 16:28:54.061018 8385 raft_consensus.cc:2229] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:54.061233 8385 raft_consensus.cc:2258] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:54.062381 8385 ts_tablet_manager.cc:1905] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:54.065847 8386 tablet_replica.cc:330] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: stopping tablet replica
I20260929 16:28:54.066082 8386 raft_consensus.cc:2229] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:54.066381 8386 raft_consensus.cc:2258] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:54.067395 8386 ts_tablet_manager.cc:1905] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:28:54 dist-test-slave-dssx krb5kdc[7755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699334, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 16:28:54.159400 6050 mini_kdc.cc:317] Time spent kinit for test-user: real 0.108s user 0.006s sys 0.006s
I20260929 16:28:54.163480 8387 ts_tablet_manager.cc:1918] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:28:54.163509 8385 ts_tablet_manager.cc:1918] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:28:54.163646 8385 log.cc:1196] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/2d37b9a5646844a3b77faf86860ae90f
I20260929 16:28:54.163837 8387 log.cc:1196] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/2d37b9a5646844a3b77faf86860ae90f
I20260929 16:28:54.164140 8385 ts_tablet_manager.cc:1939] T 2d37b9a5646844a3b77faf86860ae90f P 9d2243e6db114eda9fb2fbe25c468b17: Deleting consensus metadata
I20260929 16:28:54.164299 8387 ts_tablet_manager.cc:1939] T 2d37b9a5646844a3b77faf86860ae90f P 279a707487664029b87b0ffbc5411148: Deleting consensus metadata
Sep 29 16:28:54 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699334, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:54.190531 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:54.169827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.5.232.190:36861)
0929 16:28:54.170423 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:54.170427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:54.170454 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:28:54.170631 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:54.170635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:54.170695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:54.170788 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:54.172066 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.172871 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.174242 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.174649 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.174745 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:54.185471 (+ 10726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:54.185476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:54.185480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:54.185518 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:54.187908 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:54.188834 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:54.188838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:54.188841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:54.188973 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:54.189271 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:54.189274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:54.189276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:54.189567 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:54.189740 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:54.189977 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 16:28:54.190141 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":326,"threads_started":1}
I20260929 16:28:54.193971 7798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:28:54.194630 7798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:54.200984 8076 tablet_service.cc:1467] Processing CreateTablet for tablet 85742f224f2941108c00d4a3e8d852a4 (DEFAULT_TABLE table=test-table [id=0339b1c2e86d44079c688337f60e71cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:54.200984 7937 tablet_service.cc:1467] Processing CreateTablet for tablet 85742f224f2941108c00d4a3e8d852a4 (DEFAULT_TABLE table=test-table [id=0339b1c2e86d44079c688337f60e71cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:54.201272 8215 tablet_service.cc:1467] Processing CreateTablet for tablet 85742f224f2941108c00d4a3e8d852a4 (DEFAULT_TABLE table=test-table [id=0339b1c2e86d44079c688337f60e71cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:54.201454 7937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85742f224f2941108c00d4a3e8d852a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:54.201718 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85742f224f2941108c00d4a3e8d852a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:54.206719 8396 tablet_bootstrap.cc:492] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: Bootstrap starting.
I20260929 16:28:54.207330 8397 tablet_bootstrap.cc:492] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: Bootstrap starting.
I20260929 16:28:54.208333 8396 tablet_bootstrap.cc:654] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:54.208680 8397 tablet_bootstrap.cc:654] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:54.210875 8396 tablet_bootstrap.cc:492] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: No bootstrap required, opened a new log
I20260929 16:28:54.211022 8396 ts_tablet_manager.cc:1397] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:28:54.211045 8397 tablet_bootstrap.cc:492] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: No bootstrap required, opened a new log
I20260929 16:28:54.211187 8397 ts_tablet_manager.cc:1397] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:28:54.211673 8396 raft_consensus.cc:348] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.211848 8396 raft_consensus.cc:374] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:54.211952 8396 raft_consensus.cc:729] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2243e6db114eda9fb2fbe25c468b17, State: Initialized, Role: FOLLOWER
I20260929 16:28:54.211879 8397 raft_consensus.cc:348] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.212155 8397 raft_consensus.cc:374] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:54.212246 8397 raft_consensus.cc:729] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279a707487664029b87b0ffbc5411148, State: Initialized, Role: FOLLOWER
I20260929 16:28:54.212227 8396 consensus_queue.cc:260] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.212487 8397 consensus_queue.cc:260] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.212865 8396 ts_tablet_manager.cc:1428] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:28:54.213122 8397 ts_tablet_manager.cc:1428] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:28:54.254143 7785 catalog_manager.cc:4841] TS 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465): tablet 2d37b9a5646844a3b77faf86860ae90f (table test-table [id=0e7b752486ac40dcaed249674ba6ff7c]) successfully deleted
I20260929 16:28:54.254176 7787 catalog_manager.cc:4841] TS 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825): tablet 2d37b9a5646844a3b77faf86860ae90f (table test-table [id=0e7b752486ac40dcaed249674ba6ff7c]) successfully deleted
I20260929 16:28:54.262346 8076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85742f224f2941108c00d4a3e8d852a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:54.262502 8386 ts_tablet_manager.cc:1918] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:28:54.262682 8386 log.cc:1196] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/2d37b9a5646844a3b77faf86860ae90f
I20260929 16:28:54.263227 8386 ts_tablet_manager.cc:1939] T 2d37b9a5646844a3b77faf86860ae90f P 499388d1f2754abc994531cbee83d442: Deleting consensus metadata
I20260929 16:28:54.264544 7786 catalog_manager.cc:4841] TS 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019): tablet 2d37b9a5646844a3b77faf86860ae90f (table test-table [id=0e7b752486ac40dcaed249674ba6ff7c]) successfully deleted
I20260929 16:28:54.268635 8400 tablet_bootstrap.cc:492] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: Bootstrap starting.
I20260929 16:28:54.270562 8400 tablet_bootstrap.cc:654] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:54.272344 8400 tablet_bootstrap.cc:492] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: No bootstrap required, opened a new log
I20260929 16:28:54.272469 8400 ts_tablet_manager.cc:1397] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 16:28:54.273116 8400 raft_consensus.cc:348] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.273321 8400 raft_consensus.cc:374] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:54.273464 8400 raft_consensus.cc:729] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499388d1f2754abc994531cbee83d442, State: Initialized, Role: FOLLOWER
I20260929 16:28:54.273706 8400 consensus_queue.cc:260] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.274498 8400 ts_tablet_manager.cc:1428] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:28:54.315645 8380 raft_consensus.cc:482] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:54.315867 8380 raft_consensus.cc:504] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.316593 8380 leader_election.cc:288] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825), 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019)
I20260929 16:28:54.317226 8096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85742f224f2941108c00d4a3e8d852a4" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442" is_pre_election: true
I20260929 16:28:54.317205 7956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85742f224f2941108c00d4a3e8d852a4" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2243e6db114eda9fb2fbe25c468b17" is_pre_election: true
I20260929 16:28:54.317510 7956 raft_consensus.cc:2454] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 0.
I20260929 16:28:54.317505 8096 raft_consensus.cc:2454] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 0.
I20260929 16:28:54.317992 8168 leader_election.cc:302] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279a707487664029b87b0ffbc5411148, 499388d1f2754abc994531cbee83d442; no voters:
I20260929 16:28:54.318286 8380 raft_consensus.cc:2781] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:54.318408 8380 raft_consensus.cc:482] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:54.318511 8380 raft_consensus.cc:3037] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:54.364980 8380 raft_consensus.cc:504] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.365808 8380 leader_election.cc:288] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 election: Requested vote from peers 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825), 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019)
I20260929 16:28:54.366436 7956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85742f224f2941108c00d4a3e8d852a4" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2243e6db114eda9fb2fbe25c468b17"
I20260929 16:28:54.366423 8096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85742f224f2941108c00d4a3e8d852a4" candidate_uuid: "279a707487664029b87b0ffbc5411148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442"
I20260929 16:28:54.366671 8096 raft_consensus.cc:3037] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:54.366671 7956 raft_consensus.cc:3037] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:54.469573 8096 raft_consensus.cc:2454] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 1.
I20260929 16:28:54.469573 7956 raft_consensus.cc:2454] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 279a707487664029b87b0ffbc5411148 in term 1.
I20260929 16:28:54.470698 8168 leader_election.cc:302] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279a707487664029b87b0ffbc5411148, 499388d1f2754abc994531cbee83d442; no voters:
I20260929 16:28:54.471117 8380 raft_consensus.cc:2781] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:54.471417 8380 raft_consensus.cc:686] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 1 LEADER]: Becoming Leader. State: Replica: 279a707487664029b87b0ffbc5411148, State: Running, Role: LEADER
I20260929 16:28:54.471760 8380 consensus_queue.cc:237] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.475011 7798 catalog_manager.cc:5495] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 reported cstate change: term changed from 0 to 1, leader changed from <none> to 279a707487664029b87b0ffbc5411148 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "279a707487664029b87b0ffbc5411148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } health_report { overall_health: HEALTHY } } }
I20260929 16:28:54.514768 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:54.509609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.5.232.131:43465)
0929 16:28:54.510256 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:54.510261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:54.510278 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:28:54.510328 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:54.510331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:54.510389 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:54.510499 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:54.510922 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.511636 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.512944 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.513370 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.513529 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:54.513970 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:54.514111 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:54.514209 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:54.514369 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 16:28:54.514446 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":353,"threads_started":1}
I20260929 16:28:54.523381 8096 raft_consensus.cc:1261] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Refusing update from remote peer 279a707487664029b87b0ffbc5411148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:54.523669 7956 raft_consensus.cc:1261] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Refusing update from remote peer 279a707487664029b87b0ffbc5411148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:54.523967 8375 consensus_queue.cc:1035] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:54.524281 8380 consensus_queue.cc:1035] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:54.598762 7798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:54.599043 7798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:54.602216 7798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018: Sending DeleteTablet for 3 replicas of tablet 85742f224f2941108c00d4a3e8d852a4
I20260929 16:28:54.603098 7937 tablet_service.cc:1514] Processing DeleteTablet for tablet 85742f224f2941108c00d4a3e8d852a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:54 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:50514
I20260929 16:28:54.603237 8076 tablet_service.cc:1514] Processing DeleteTablet for tablet 85742f224f2941108c00d4a3e8d852a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:54 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:52716
I20260929 16:28:54.603528 8385 tablet_replica.cc:330] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: stopping tablet replica
I20260929 16:28:54.603585 8386 tablet_replica.cc:330] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: stopping tablet replica
I20260929 16:28:54.603714 8385 raft_consensus.cc:2229] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:54.603793 8386 raft_consensus.cc:2229] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:54.603886 8385 raft_consensus.cc:2258] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:54.603999 8386 raft_consensus.cc:2258] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:54.604261 7798 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 16:28:54.604933 8385 ts_tablet_manager.cc:1905] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:28:54.604993 7798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:54.606125 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 85742f224f2941108c00d4a3e8d852a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:54 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:53672
I20260929 16:28:54.606405 8387 tablet_replica.cc:330] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: stopping tablet replica
I20260929 16:28:54.606590 8387 raft_consensus.cc:2229] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:28:54.606905 8387 raft_consensus.cc:2258] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:54.608132 8387 ts_tablet_manager.cc:1905] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:54.608397 8386 ts_tablet_manager.cc:1905] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:54.613153 7937 tablet_service.cc:1467] Processing CreateTablet for tablet d9afe56524624b8e8ec7e40aca657da5 (DEFAULT_TABLE table=test-table [id=9c8dd5bf02ff4da595e3cb3c77619a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:54.613499 8076 tablet_service.cc:1467] Processing CreateTablet for tablet d9afe56524624b8e8ec7e40aca657da5 (DEFAULT_TABLE table=test-table [id=9c8dd5bf02ff4da595e3cb3c77619a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:54.613970 8215 tablet_service.cc:1467] Processing CreateTablet for tablet d9afe56524624b8e8ec7e40aca657da5 (DEFAULT_TABLE table=test-table [id=9c8dd5bf02ff4da595e3cb3c77619a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:54.674465 7937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9afe56524624b8e8ec7e40aca657da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:54.674487 8076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9afe56524624b8e8ec7e40aca657da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:54.678984 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9afe56524624b8e8ec7e40aca657da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:54.682535 8400 tablet_bootstrap.cc:492] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: Bootstrap starting.
I20260929 16:28:54.682633 8396 tablet_bootstrap.cc:492] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17: Bootstrap starting.
I20260929 16:28:54.684437 8396 tablet_bootstrap.cc:654] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:54.684473 8400 tablet_bootstrap.cc:654] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:54.686947 8396 tablet_bootstrap.cc:492] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17: No bootstrap required, opened a new log
I20260929 16:28:54.687155 8400 tablet_bootstrap.cc:492] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: No bootstrap required, opened a new log
I20260929 16:28:54.687163 8396 ts_tablet_manager.cc:1397] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:28:54.687328 8400 ts_tablet_manager.cc:1397] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 16:28:54.687860 8396 raft_consensus.cc:348] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.687914 8400 raft_consensus.cc:348] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.688083 8396 raft_consensus.cc:374] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:54.688118 8400 raft_consensus.cc:374] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:54.688182 8396 raft_consensus.cc:729] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2243e6db114eda9fb2fbe25c468b17, State: Initialized, Role: FOLLOWER
I20260929 16:28:54.688208 8400 raft_consensus.cc:729] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499388d1f2754abc994531cbee83d442, State: Initialized, Role: FOLLOWER
I20260929 16:28:54.688446 8396 consensus_queue.cc:260] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.688462 8400 consensus_queue.cc:260] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.688944 8400 ts_tablet_manager.cc:1428] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:28:54.689551 8396 ts_tablet_manager.cc:1428] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:28:54.711740 8327 raft_consensus.cc:482] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:54.712095 8327 raft_consensus.cc:504] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.712738 8327 leader_election.cc:288] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019), 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465)
I20260929 16:28:54.713425 8096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9afe56524624b8e8ec7e40aca657da5" candidate_uuid: "9d2243e6db114eda9fb2fbe25c468b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442" is_pre_election: true
I20260929 16:28:54.713675 8096 raft_consensus.cc:2454] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2243e6db114eda9fb2fbe25c468b17 in term 0.
I20260929 16:28:54.714063 7889 leader_election.cc:302] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499388d1f2754abc994531cbee83d442, 9d2243e6db114eda9fb2fbe25c468b17; no voters:
W20260929 16:28:54.714320 7888 leader_election.cc:341] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465): Not found: Tablet not found: d9afe56524624b8e8ec7e40aca657da5
I20260929 16:28:54.714401 8327 raft_consensus.cc:2781] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:54.714541 8327 raft_consensus.cc:482] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:54.714672 8327 raft_consensus.cc:3037] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:54.774987 8387 ts_tablet_manager.cc:1918] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:28:54.774987 8385 ts_tablet_manager.cc:1918] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:28:54.774996 8386 ts_tablet_manager.cc:1918] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:28:54.775166 8387 log.cc:1196] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/85742f224f2941108c00d4a3e8d852a4
I20260929 16:28:54.775175 8386 log.cc:1196] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/85742f224f2941108c00d4a3e8d852a4
I20260929 16:28:54.775180 8385 log.cc:1196] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/85742f224f2941108c00d4a3e8d852a4
I20260929 16:28:54.775631 8386 ts_tablet_manager.cc:1939] T 85742f224f2941108c00d4a3e8d852a4 P 499388d1f2754abc994531cbee83d442: Deleting consensus metadata
I20260929 16:28:54.775635 8387 ts_tablet_manager.cc:1939] T 85742f224f2941108c00d4a3e8d852a4 P 279a707487664029b87b0ffbc5411148: Deleting consensus metadata
I20260929 16:28:54.775643 8385 ts_tablet_manager.cc:1939] T 85742f224f2941108c00d4a3e8d852a4 P 9d2243e6db114eda9fb2fbe25c468b17: Deleting consensus metadata
I20260929 16:28:54.776852 7785 catalog_manager.cc:4841] TS 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465): tablet 85742f224f2941108c00d4a3e8d852a4 (table test-table [id=0339b1c2e86d44079c688337f60e71cd]) successfully deleted
I20260929 16:28:54.777057 7786 catalog_manager.cc:4841] TS 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019): tablet 85742f224f2941108c00d4a3e8d852a4 (table test-table [id=0339b1c2e86d44079c688337f60e71cd]) successfully deleted
I20260929 16:28:54.777514 8327 raft_consensus.cc:504] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.778344 7787 catalog_manager.cc:4841] TS 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825): tablet 85742f224f2941108c00d4a3e8d852a4 (table test-table [id=0339b1c2e86d44079c688337f60e71cd]) successfully deleted
I20260929 16:28:54.778796 8327 leader_election.cc:288] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 election: Requested vote from peers 499388d1f2754abc994531cbee83d442 (127.5.232.130:42019), 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465)
I20260929 16:28:54.779143 8096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9afe56524624b8e8ec7e40aca657da5" candidate_uuid: "9d2243e6db114eda9fb2fbe25c468b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499388d1f2754abc994531cbee83d442"
I20260929 16:28:54.779363 8096 raft_consensus.cc:3037] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:54.779349 8239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9afe56524624b8e8ec7e40aca657da5" candidate_uuid: "9d2243e6db114eda9fb2fbe25c468b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279a707487664029b87b0ffbc5411148"
I20260929 16:28:54.779640 8413 tablet_bootstrap.cc:492] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: Bootstrap starting.
W20260929 16:28:54.779937 7888 leader_election.cc:341] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465): Illegal state: must be running to vote when last-logged opid is not known
I20260929 16:28:54.781577 8413 tablet_bootstrap.cc:654] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:54.783264 8413 tablet_bootstrap.cc:492] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: No bootstrap required, opened a new log
I20260929 16:28:54.783406 8413 ts_tablet_manager.cc:1397] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 16:28:54.784080 8413 raft_consensus.cc:348] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.784257 8413 raft_consensus.cc:374] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:54.784368 8413 raft_consensus.cc:729] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279a707487664029b87b0ffbc5411148, State: Initialized, Role: FOLLOWER
I20260929 16:28:54.784617 8413 consensus_queue.cc:260] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.785110 8413 ts_tablet_manager.cc:1428] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:28:54.875052 8096 raft_consensus.cc:2454] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2243e6db114eda9fb2fbe25c468b17 in term 1.
I20260929 16:28:54.875706 7889 leader_election.cc:302] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 499388d1f2754abc994531cbee83d442, 9d2243e6db114eda9fb2fbe25c468b17; no voters: 279a707487664029b87b0ffbc5411148
I20260929 16:28:54.876106 8327 raft_consensus.cc:2781] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:54.876336 8327 raft_consensus.cc:686] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 LEADER]: Becoming Leader. State: Replica: 9d2243e6db114eda9fb2fbe25c468b17, State: Running, Role: LEADER
I20260929 16:28:54.876622 8327 consensus_queue.cc:237] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } }
I20260929 16:28:54.880332 7798 catalog_manager.cc:5495] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129). New cstate: current_term: 1 leader_uuid: "9d2243e6db114eda9fb2fbe25c468b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d2243e6db114eda9fb2fbe25c468b17" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 } health_report { overall_health: UNKNOWN } } }
I20260929 16:28:54.931135 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:54.926100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.5.232.129:46825)
0929 16:28:54.926631 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:54.926635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:54.926654 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:28:54.926704 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:54.926708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:54.926766 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:54.926892 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:54.927303 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.928048 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.929344 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.929804 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.929943 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:54.930384 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:54.930495 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:54.930592 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:54.930739 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 16:28:54.930811 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":299,"threads_started":1}
I20260929 16:28:54.945818 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:54.939748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:55633 (local address 127.5.232.129:46825)
0929 16:28:54.939943 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:54.939950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:54.940548 (+ 598us) server_negotiation.cc:407] Connection header received
0929 16:28:54.940745 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:54.940748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:54.940815 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:54.940929 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:54.941451 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.942254 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.944234 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.944612 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.944701 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:54.945063 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:54.945152 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:54.945215 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:54.945330 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 16:28:54.945463 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:28:54.945907 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:54.939762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.5.232.129:55633)
0929 16:28:54.940439 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:54.940463 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:54.940597 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:54.940947 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:54.940952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:54.940964 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:28:54.941243 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:54.941252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.942794 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.942800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:54.944115 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:54.944127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:54.944746 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:54.944750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:54.944864 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:54.944897 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:28:54.945232 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:28:54.945239 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:28:54.945298 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 16:28:54.945503 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":68,"thread_start_us":371,"threads_started":1}
I20260929 16:28:54.953581 8239 raft_consensus.cc:3037] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:54.953734 8096 raft_consensus.cc:1261] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Refusing update from remote peer 9d2243e6db114eda9fb2fbe25c468b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:54.954172 8379 consensus_queue.cc:1035] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "499388d1f2754abc994531cbee83d442" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:28:54.971632 8364 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:28:54.976593 8239 raft_consensus.cc:1261] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Refusing update from remote peer 9d2243e6db114eda9fb2fbe25c468b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:54.979889 8327 consensus_queue.cc:1035] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "279a707487664029b87b0ffbc5411148" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:56.587532 7800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:56.587810 7800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:56.592669 7800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deabc511b35a452e9d2834006cbd5018: Sending DeleteTablet for 3 replicas of tablet d9afe56524624b8e8ec7e40aca657da5
I20260929 16:28:56.593667 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet d9afe56524624b8e8ec7e40aca657da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:56 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:53672
I20260929 16:28:56.593679 7936 tablet_service.cc:1514] Processing DeleteTablet for tablet d9afe56524624b8e8ec7e40aca657da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:56 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:50514
I20260929 16:28:56.593707 8076 tablet_service.cc:1514] Processing DeleteTablet for tablet d9afe56524624b8e8ec7e40aca657da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:56 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:52716
I20260929 16:28:56.594555 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7871
I20260929 16:28:56.606562 8444 tablet_replica.cc:330] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: stopping tablet replica
I20260929 16:28:56.606963 8444 raft_consensus.cc:2229] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:56.607350 8444 raft_consensus.cc:2258] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:56.608961 8444 ts_tablet_manager.cc:1905] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:56.614015 8446 tablet_replica.cc:330] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17: stopping tablet replica
I20260929 16:28:56.614228 8445 tablet_replica.cc:330] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: stopping tablet replica
I20260929 16:28:56.614420 8446 raft_consensus.cc:2229] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:28:56.614527 8445 raft_consensus.cc:2229] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:56.614840 8445 raft_consensus.cc:2258] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:56.615638 8446 raft_consensus.cc:2258] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:56.616451 8445 ts_tablet_manager.cc:1905] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:56.617481 8446 ts_tablet_manager.cc:1905] T d9afe56524624b8e8ec7e40aca657da5 P 9d2243e6db114eda9fb2fbe25c468b17: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:28:56 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699334, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:28:56.633630 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.598715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.5.232.129:46825)
0929 16:28:56.616208 (+ 17493us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:56.616214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:56.616239 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:28:56.616306 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:56.616314 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:56.616398 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:56.616511 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:56.618455 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:56.619592 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:56.621331 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:56.622014 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:56.622147 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:56.627222 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:56.627239 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:56.627258 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:56.627297 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:56.630298 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:56.631003 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:56.631010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:56.631013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:56.631148 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:56.631491 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:56.631494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:56.631496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:56.631839 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:56.632065 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:56.632325 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:28:56.632496 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17318,"thread_start_us":17162,"threads_started":1}
W20260929 16:28:56.634321 8002 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60318
W20260929 16:28:56.635002 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7871: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:28:56.635208 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 7871
I20260929 16:28:56.690078 8445 ts_tablet_manager.cc:1918] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:28:56.690092 8444 ts_tablet_manager.cc:1918] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:28:56.690270 8445 log.cc:1196] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/d9afe56524624b8e8ec7e40aca657da5
I20260929 16:28:56.690279 8444 log.cc:1196] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/d9afe56524624b8e8ec7e40aca657da5
I20260929 16:28:56.690696 8445 ts_tablet_manager.cc:1939] T d9afe56524624b8e8ec7e40aca657da5 P 499388d1f2754abc994531cbee83d442: Deleting consensus metadata
I20260929 16:28:56.690719 8444 ts_tablet_manager.cc:1939] T d9afe56524624b8e8ec7e40aca657da5 P 279a707487664029b87b0ffbc5411148: Deleting consensus metadata
W20260929 16:28:56.711503 7787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.232.129:46825 (error 108)
I20260929 16:28:56.712005 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8011
I20260929 16:28:56.714033 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.712876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.0.0.1:60330)
0929 16:28:56.713599 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.713792 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":430,"threads_started":1}
W20260929 16:28:56.714282 7787 catalog_manager.cc:4584] TS 9d2243e6db114eda9fb2fbe25c468b17 (127.5.232.129:46825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9afe56524624b8e8ec7e40aca657da5: Network error: Client connection negotiation failed: client connection to 127.5.232.129:46825: connect: Connection refused (error 111)
Sep 29 16:28:56 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699334, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:28:56.727321 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.712809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.5.232.130:42019)
0929 16:28:56.713369 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:56.713397 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:56.713419 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:28:56.713489 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:56.713494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:56.713575 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:56.713705 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:56.715041 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:56.715796 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:56.716972 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:56.717373 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:56.717555 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:56.721781 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:56.721796 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:56.721813 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:56.721853 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:56.724589 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:56.725299 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:56.725306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:56.725309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:56.725553 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:56.725896 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:56.725899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:56.725901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:56.726253 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:56.726477 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:56.726732 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 16:28:56.726908 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":290,"threads_started":1}
W20260929 16:28:56.728147 8140 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39084
W20260929 16:28:56.728813 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8011: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:28:56.729013 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 8011
I20260929 16:28:56.768337 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.767867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.0.0.1:60334)
0929 16:28:56.768097 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.768188 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 16:28:56.769008 7785 catalog_manager.cc:4841] TS 279a707487664029b87b0ffbc5411148 (127.5.232.131:43465): tablet d9afe56524624b8e8ec7e40aca657da5 (table test-table [id=9c8dd5bf02ff4da595e3cb3c77619a03]) successfully deleted
I20260929 16:28:56.769572 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.769084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.0.0.1:60342)
0929 16:28:56.769268 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.769354 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:28:56.791783 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8150
I20260929 16:28:56.792906 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.792362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:42019 (local address 127.0.0.1:39090)
0929 16:28:56.792596 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.792690 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.130:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
Sep 29 16:28:56 dist-test-slave-dssx krb5kdc[7755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699334, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:28:56.807726 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.792605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.5.232.131:43465)
0929 16:28:56.793167 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:56.793171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:56.793187 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:28:56.793238 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:56.793242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:56.793318 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:56.793479 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:56.794810 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:56.795570 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:56.796778 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:56.797187 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:56.797327 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:56.802125 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:56.802146 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:56.802162 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:56.802201 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:56.805124 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:56.805881 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:56.805888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:56.805890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:56.806028 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:56.806501 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:56.806505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:56.806507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:56.806865 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:56.807096 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:56.807298 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 16:28:56.807463 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":258,"threads_started":1}
W20260929 16:28:56.808384 8280 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51370
W20260929 16:28:56.808961 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8150: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:28:56.809094 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 8150
I20260929 16:28:56.825296 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.824850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:42019 (local address 127.0.0.1:39094)
0929 16:28:56.825054 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.825139 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.130:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:28:56.826388 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.826045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:42019 (local address 127.0.0.1:39106)
0929 16:28:56.826212 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.826279 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.130:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 16:28:56.829705 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.829300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.0.0.1:60348)
0929 16:28:56.829498 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.829562 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 16:28:56.830678 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.830345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.0.0.1:60364)
0929 16:28:56.830510 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.830582 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:28:56.831493 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 7768
I20260929 16:28:56.832578 7840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:28:56.952042 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.951106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:46825 (local address 127.0.0.1:60370)
0929 16:28:56.951465 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.951569 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260929 16:28:56.953186 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 7768
2026-09-29T16:28:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8769 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:28:58 dist-test-slave-dssx krb5kdc[8457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:28:58 dist-test-slave-dssx krb5kdc[8457](info): set up 2 sockets
Sep 29 16:28:58 dist-test-slave-dssx krb5kdc[8457](info): commencing operation
krb5kdc: starting...
W20260929 16:28:59.225863 6050 mini_kdc.cc:121] Time spent starting KDC: real 2.228s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:28:59 dist-test-slave-dssx krb5kdc[8457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:28:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:28:59.416944 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:35835
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:59.770244 8470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:59.770627 8470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:59.770793 8470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:59.778061 8470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:59.778211 8470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:59.778318 8470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:59.778421 8470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:59.778508 8470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:59.788425 8470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:35835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:28:59.789898 8470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:59.791289 8470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:59.803901 8475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:59.803854 8476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:59.803984 8470 server_base.cc:1034] running on GCE node
W20260929 16:28:59.804095 8478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:59.805122 8470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:59.806685 8470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:59.807948 8470 hybrid_clock.cc:648] HybridClock initialized: now 1790699339807872 us; error 62 us; skew 500 ppm
I20260929 16:28:59.808368 8470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:28:59 dist-test-slave-dssx krb5kdc[8457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699339, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:59.815773 8470 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:28:59.817345 8470 webserver.cc:460] Webserver started at http://127.5.232.190:34483/ using document root <none> and password file <none>
I20260929 16:28:59.818168 8470 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:59.818317 8470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:59.818652 8470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:59.821803 8470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "2077106216e44141a1423e7bae0b16f0"
format_stamp: "Formatted at 2026-09-29 16:28:59 on dist-test-slave-dssx"
server_key: "6e57b1f9bb041611a906a215ec8a4dc9"
server_key_iv: "40dd9c4eaddc8802b3f7d6076a7330af"
server_key_version: "encryptionkey@0"
I20260929 16:28:59.822654 8470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "2077106216e44141a1423e7bae0b16f0"
format_stamp: "Formatted at 2026-09-29 16:28:59 on dist-test-slave-dssx"
server_key: "6e57b1f9bb041611a906a215ec8a4dc9"
server_key_iv: "40dd9c4eaddc8802b3f7d6076a7330af"
server_key_version: "encryptionkey@0"
I20260929 16:28:59.827951 8470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:28:59.831573 8485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:59.832967 8470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:28:59.833178 8470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "2077106216e44141a1423e7bae0b16f0"
format_stamp: "Formatted at 2026-09-29 16:28:59 on dist-test-slave-dssx"
server_key: "6e57b1f9bb041611a906a215ec8a4dc9"
server_key_iv: "40dd9c4eaddc8802b3f7d6076a7330af"
server_key_version: "encryptionkey@0"
I20260929 16:28:59.833386 8470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:59.861924 8470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:59.865916 8470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:59.866241 8470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:59.884517 8470 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:35835
I20260929 16:28:59.884523 8536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:35835 every 8 connection(s)
I20260929 16:28:59.886070 8470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:59.890179 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:59.896230 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 8470
I20260929 16:28:59.896471 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:28:59.897289 6050 external_mini_cluster.cc:1428] Setting key 447d9bd3912e3c3b832c883fc6a067e3
I20260929 16:28:59.902463 8537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0: Bootstrap starting.
I20260929 16:28:59.905802 8537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:59.906909 8537 log.cc:824] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:59.910311 8537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0: No bootstrap required, opened a new log
Sep 29 16:28:59 dist-test-slave-dssx krb5kdc[8457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:28:59.920665 8537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2077106216e44141a1423e7bae0b16f0" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35835 } }
I20260929 16:28:59.921142 8537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:59.921252 8537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2077106216e44141a1423e7bae0b16f0, State: Initialized, Role: FOLLOWER
I20260929 16:28:59.921998 8537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2077106216e44141a1423e7bae0b16f0" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35835 } }
I20260929 16:28:59.922293 8537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:59.922434 8537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:59.922657 8537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:59.922694 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:59.899628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.5.232.190:35835)
0929 16:28:59.900514 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:59.900527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:59.900590 (+ 63us) server_negotiation.cc:407] Connection header received
0929 16:28:59.902418 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:59.902448 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:59.902965 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:59.903388 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:59.904885 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:59.906355 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:59.907919 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:59.908879 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:59.909081 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:59.913542 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:59.913557 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:59.913578 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:59.913629 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:59.917783 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:59.918531 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:59.918544 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:59.918559 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:59.918791 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:59.919254 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:59.919258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:59.919262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:59.920029 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:59.920314 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:59.920702 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 16:28:59.921164 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":428,"threads_started":1}
I20260929 16:28:59.925053 8537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2077106216e44141a1423e7bae0b16f0" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35835 } }
I20260929 16:28:59.925661 8537 leader_election.cc:302] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2077106216e44141a1423e7bae0b16f0; no voters:
I20260929 16:28:59.926245 8537 leader_election.cc:288] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:28:59.926398 8542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:59.927158 8542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [term 1 LEADER]: Becoming Leader. State: Replica: 2077106216e44141a1423e7bae0b16f0, State: Running, Role: LEADER
I20260929 16:28:59.927729 8542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2077106216e44141a1423e7bae0b16f0" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35835 } }
I20260929 16:28:59.928620 8537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:59.930934 8544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2077106216e44141a1423e7bae0b16f0. Latest consensus state: current_term: 1 leader_uuid: "2077106216e44141a1423e7bae0b16f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2077106216e44141a1423e7bae0b16f0" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35835 } } }
I20260929 16:28:59.931295 8544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:59.930717 8543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2077106216e44141a1423e7bae0b16f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2077106216e44141a1423e7bae0b16f0" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35835 } } }
I20260929 16:28:59.931910 8543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0 [sys.catalog]: This master's current role is: LEADER
W20260929 16:28:59.942028 8558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:28:59.942203 8558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:28:59.942368 8549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:59.944414 8549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:59.953483 8549 catalog_manager.cc:1347] Generated new cluster ID: 25295155ead345378ba6c7d8cff3741c
I20260929 16:28:59.953639 8549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:59.992199 8549 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:28:59.993599 8549 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:00.002998 8549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2077106216e44141a1423e7bae0b16f0: Generated new TSK 0
I20260929 16:29:00.003909 8549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:29:00.058487 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35835
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:00.416858 8563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:00.417169 8563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:00.417266 8563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:00.425019 8563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:00.425148 8563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:00.425266 8563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:00.435835 8563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:00.437337 8563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:00.438877 8563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:00.452334 8568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:00.452363 8571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:00.452399 8569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:00.453229 8563 server_base.cc:1034] running on GCE node
I20260929 16:29:00.453946 8563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:00.455021 8563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:00.456276 8563 hybrid_clock.cc:648] HybridClock initialized: now 1790699340456222 us; error 51 us; skew 500 ppm
I20260929 16:29:00.456632 8563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:00 dist-test-slave-dssx krb5kdc[8457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699340, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:00.459973 8563 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:29:00.461647 8563 webserver.cc:460] Webserver started at http://127.5.232.129:39277/ using document root <none> and password file <none>
I20260929 16:29:00.462374 8563 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:00.462524 8563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:00.462869 8563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:00.465991 8563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "7338a761023b4554a9228e73acf26077"
format_stamp: "Formatted at 2026-09-29 16:29:00 on dist-test-slave-dssx"
server_key: "1a68e7e6ccb2a72ebfe22daed52461cb"
server_key_iv: "e56bd23016a502b5fdc0318690381291"
server_key_version: "encryptionkey@0"
I20260929 16:29:00.466843 8563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "7338a761023b4554a9228e73acf26077"
format_stamp: "Formatted at 2026-09-29 16:29:00 on dist-test-slave-dssx"
server_key: "1a68e7e6ccb2a72ebfe22daed52461cb"
server_key_iv: "e56bd23016a502b5fdc0318690381291"
server_key_version: "encryptionkey@0"
I20260929 16:29:00.472396 8563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:29:00.476141 8578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:00.477756 8563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:29:00.477943 8563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "7338a761023b4554a9228e73acf26077"
format_stamp: "Formatted at 2026-09-29 16:29:00 on dist-test-slave-dssx"
server_key: "1a68e7e6ccb2a72ebfe22daed52461cb"
server_key_iv: "e56bd23016a502b5fdc0318690381291"
server_key_version: "encryptionkey@0"
I20260929 16:29:00.478158 8563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:00.494148 8563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:00.498299 8563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:00.498608 8563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:00.499603 8563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:00.500978 8563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:00.501082 8563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:00.501235 8563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:00.501317 8563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:00.537596 8563 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:38465
I20260929 16:29:00.537628 8690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:38465 every 8 connection(s)
I20260929 16:29:00.539091 8563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:00.540845 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 8563
I20260929 16:29:00.541081 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:00.542019 6050 external_mini_cluster.cc:1428] Setting key 3042cdcce6988d0495c80784ff0e4be1
Sep 29 16:29:00 dist-test-slave-dssx krb5kdc[8457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699340, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
I20260929 16:29:00.561342 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:00.542173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:49383 (local address 127.5.232.190:35835)
0929 16:29:00.543469 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:00.543480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:00.544215 (+ 735us) server_negotiation.cc:407] Connection header received
0929 16:29:00.546252 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:00.546258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:00.546347 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:00.546472 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:00.548113 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:00.548831 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:00.550592 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:00.551006 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:00.551152 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:00.555268 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:00.555273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:00.555277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:00.555313 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:00.558123 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:00.558955 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:00.558960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:00.558962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:00.559080 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:00.559540 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:00.559550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:00.559552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:00.559820 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:00.559975 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:00.560731 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 16:29:00.560970 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":984,"threads_started":1}
I20260929 16:29:00.562435 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:00.543051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35835 (local address 127.5.232.129:49383)
0929 16:29:00.544052 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:00.544094 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:00.545915 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:00.546747 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:00.546756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:00.547314 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:00.547923 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:00.547942 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:00.549021 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:00.549025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:00.550416 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:00.550429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:00.551254 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:00.551263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:00.551528 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:00.552312 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:00.552335 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:00.554990 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:00.558308 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:00.558325 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:00.558341 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:00.558798 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:00.559202 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:00.559206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:00.559210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:00.559412 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:00.559951 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:00.559960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:00.560341 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 16:29:00.560624 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 16:29:00.561197 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":358,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
I20260929 16:29:00.564291 8691 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35835
I20260929 16:29:00.564652 8691 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:00.565584 8691 heartbeater.cc:511] Master 127.5.232.190:35835 requested a full tablet report, sending...
I20260929 16:29:00.568179 8502 ts_manager.cc:194] Registered new tserver with Master: 7338a761023b4554a9228e73acf26077 (127.5.232.129:38465)
I20260929 16:29:00.570163 8502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:49383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:29:00.588835 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35835
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:00.921701 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:00.922042 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:00.922230 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:00.929682 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:00.929836 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:00.930006 8697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:00.940244 8697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:00.941725 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:00.943246 8697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:00.956826 8702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:00.956873 8705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:00.956839 8703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:00.957978 8697 server_base.cc:1034] running on GCE node
I20260929 16:29:00.958643 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:00.959617 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:00.960827 8697 hybrid_clock.cc:648] HybridClock initialized: now 1790699340960760 us; error 58 us; skew 500 ppm
I20260929 16:29:00.961158 8697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:00 dist-test-slave-dssx krb5kdc[8457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699340, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:00.964385 8697 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:29:00.965968 8697 webserver.cc:460] Webserver started at http://127.5.232.130:38505/ using document root <none> and password file <none>
I20260929 16:29:00.966764 8697 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:00.966910 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:00.967223 8697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:00.970259 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "afc15225f4aa47efb16bf6c36facf693"
format_stamp: "Formatted at 2026-09-29 16:29:00 on dist-test-slave-dssx"
server_key: "3bab373238430387aa28df73fba7590d"
server_key_iv: "56102b233fbcfa12389e4c529771bda0"
server_key_version: "encryptionkey@0"
I20260929 16:29:00.971077 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "afc15225f4aa47efb16bf6c36facf693"
format_stamp: "Formatted at 2026-09-29 16:29:00 on dist-test-slave-dssx"
server_key: "3bab373238430387aa28df73fba7590d"
server_key_iv: "56102b233fbcfa12389e4c529771bda0"
server_key_version: "encryptionkey@0"
I20260929 16:29:00.976595 8697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 16:29:00.980446 8712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:00.982192 8697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:00.982379 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "afc15225f4aa47efb16bf6c36facf693"
format_stamp: "Formatted at 2026-09-29 16:29:00 on dist-test-slave-dssx"
server_key: "3bab373238430387aa28df73fba7590d"
server_key_iv: "56102b233fbcfa12389e4c529771bda0"
server_key_version: "encryptionkey@0"
I20260929 16:29:00.982618 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:00.996848 8697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:01.001471 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:01.001857 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:01.003123 8697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:01.004580 8697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:01.004685 8697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:01.004839 8697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:01.004920 8697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:01.039260 8697 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:43047
I20260929 16:29:01.039310 8824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:43047 every 8 connection(s)
I20260929 16:29:01.040657 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:01.047995 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 8697
I20260929 16:29:01.048168 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:01.048643 6050 external_mini_cluster.cc:1428] Setting key 11811d18126929ad8002f559d18d7327
Sep 29 16:29:01 dist-test-slave-dssx krb5kdc[8457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699340, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:01.063889 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:01.043430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:47957 (local address 127.5.232.190:35835)
0929 16:29:01.043642 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:01.043649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:01.044900 (+ 1251us) server_negotiation.cc:407] Connection header received
0929 16:29:01.046788 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:01.046792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:01.046865 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:01.046960 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:01.048744 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:01.049561 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:01.051606 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:01.052416 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:01.052575 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:01.056429 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:01.056439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:01.056445 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:01.056491 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:01.060271 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:01.061107 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:01.061115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:01.061121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:01.061283 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:01.061797 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:01.061805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:01.061811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:01.062209 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:01.062370 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:01.063161 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 16:29:01.063450 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:29:01.064699 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:01.043994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35835 (local address 127.5.232.130:47957)
0929 16:29:01.044724 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:01.044768 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:01.046488 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:01.047266 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:01.047276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:01.047835 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:01.048545 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:01.048564 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:01.049745 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:01.049750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:01.051440 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:01.051454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:01.052637 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:01.052641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:01.052852 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:01.053660 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:01.053684 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:01.056150 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:01.060456 (+ 4306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:01.060465 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:01.060481 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:01.060970 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:01.061462 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:01.061465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:01.061468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:01.061672 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:01.062349 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:01.062356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:01.062736 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 16:29:01.063012 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 16:29:01.063519 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":282,"threads_started":1}
I20260929 16:29:01.066367 8825 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35835
I20260929 16:29:01.066720 8825 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:01.067469 8825 heartbeater.cc:511] Master 127.5.232.190:35835 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
I20260929 16:29:01.069001 8502 ts_manager.cc:194] Registered new tserver with Master: afc15225f4aa47efb16bf6c36facf693 (127.5.232.130:43047)
I20260929 16:29:01.069948 8502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:47957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:29:01.094607 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35835
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:01.426535 8830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:01.426839 8830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:01.427016 8830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:01.434878 8830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:01.435024 8830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:01.435221 8830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:01.445614 8830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:01.446967 8830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:01.448428 8830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:01.461874 8838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:01.461838 8836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:01.461935 8835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:01.462944 8830 server_base.cc:1034] running on GCE node
I20260929 16:29:01.463562 8830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:01.464548 8830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:01.465788 8830 hybrid_clock.cc:648] HybridClock initialized: now 1790699341465725 us; error 55 us; skew 500 ppm
I20260929 16:29:01.466130 8830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:01 dist-test-slave-dssx krb5kdc[8457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699341, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:01.469672 8830 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:29:01.471217 8830 webserver.cc:460] Webserver started at http://127.5.232.131:45173/ using document root <none> and password file <none>
I20260929 16:29:01.471968 8830 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:01.472118 8830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:01.472440 8830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:01.475477 8830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "3a0571f89dfb4676aba9b1bb1ec99796"
format_stamp: "Formatted at 2026-09-29 16:29:01 on dist-test-slave-dssx"
server_key: "cec45b121b6a16e9f636726ae8b1c3f3"
server_key_iv: "ff40cb6139378ae90f4393e46aa6d23e"
server_key_version: "encryptionkey@0"
I20260929 16:29:01.476310 8830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "3a0571f89dfb4676aba9b1bb1ec99796"
format_stamp: "Formatted at 2026-09-29 16:29:01 on dist-test-slave-dssx"
server_key: "cec45b121b6a16e9f636726ae8b1c3f3"
server_key_iv: "ff40cb6139378ae90f4393e46aa6d23e"
server_key_version: "encryptionkey@0"
I20260929 16:29:01.481761 8830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:29:01.485634 8845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:01.487144 8830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:01.487317 8830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "3a0571f89dfb4676aba9b1bb1ec99796"
format_stamp: "Formatted at 2026-09-29 16:29:01 on dist-test-slave-dssx"
server_key: "cec45b121b6a16e9f636726ae8b1c3f3"
server_key_iv: "ff40cb6139378ae90f4393e46aa6d23e"
server_key_version: "encryptionkey@0"
I20260929 16:29:01.487525 8830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:01.511324 8830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:01.515841 8830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:01.516209 8830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:01.517241 8830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:01.518734 8830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:01.518843 8830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:01.518998 8830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:01.519070 8830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:01.552825 8830 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:41389
I20260929 16:29:01.552874 8957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:41389 every 8 connection(s)
I20260929 16:29:01.554323 8830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:01.564354 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 8830
I20260929 16:29:01.564550 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:01.565060 6050 external_mini_cluster.cc:1428] Setting key e4ee713831403cc3dc1c5840c29be9d9
Sep 29 16:29:01 dist-test-slave-dssx krb5kdc[8457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699341, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:01.574395 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:01.556884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:51855 (local address 127.5.232.190:35835)
0929 16:29:01.557101 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:01.557110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:01.558358 (+ 1248us) server_negotiation.cc:407] Connection header received
0929 16:29:01.560114 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:01.560120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:01.560201 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:01.560293 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:01.561865 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:01.562590 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:01.564289 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:01.564798 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:01.564896 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:01.568909 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:01.568914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:01.568917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:01.568950 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:01.571532 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:01.572206 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:01.572210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:01.572212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:01.572334 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:01.572758 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:01.572762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:01.572764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:01.573041 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:01.573168 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:01.573893 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 16:29:01.574053 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:29:01.574422 8691 heartbeater.cc:503] Master 127.5.232.190:35835 was elected leader, sending a full tablet report...
I20260929 16:29:01.575464 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:01.557480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35835 (local address 127.5.232.131:51855)
0929 16:29:01.558190 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:01.558229 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:01.559854 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:01.560504 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:01.560512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:01.561042 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:01.561706 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:01.561728 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:01.562748 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:01.562752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:01.564136 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:01.564153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:01.565025 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:01.565035 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:01.565419 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:01.566217 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:01.566240 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:01.568705 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:01.571671 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:01.571678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:01.571692 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:01.572098 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:01.572422 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:01.572425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:01.572427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:01.572597 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:01.573148 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:01.573154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:01.573548 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 16:29:01.573792 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 16:29:01.574171 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":264,"threads_started":1}
I20260929 16:29:01.576922 8958 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35835
I20260929 16:29:01.577222 8958 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:01.577996 8958 heartbeater.cc:511] Master 127.5.232.190:35835 requested a full tablet report, sending...
I20260929 16:29:01.579475 8501 ts_manager.cc:194] Registered new tserver with Master: 3a0571f89dfb4676aba9b1bb1ec99796 (127.5.232.131:41389)
I20260929 16:29:01.580528 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:29:01.580654 8501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:51855
W20260929 16:29:01.611496 8693 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:01.608914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.5.232.190:35835)
0929 16:29:01.609157 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:01.609165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:01.609614 (+ 449us) server_negotiation.cc:407] Connection header received
0929 16:29:01.609772 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:01.609777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:01.609842 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:01.609933 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:01.611228 (+ 1295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35454: BlockingRecv error: recv got EOF from 127.0.0.1:35454 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:29:01.614825 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8563
W20260929 16:29:01.617736 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8563: Not authorized: Client connection negotiation failed: client connection to 127.5.232.129:38465: server requires authentication, but client does not have Kerberos credentials available
I20260929 16:29:01.617954 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 8563
I20260929 16:29:01.633812 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8697
W20260929 16:29:01.636724 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8697: Not authorized: Client connection negotiation failed: client connection to 127.5.232.130:43047: server requires authentication, but client does not have Kerberos credentials available
I20260929 16:29:01.636895 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 8697
W20260929 16:29:01.636951 8970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:01.634809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.5.232.130:43047)
0929 16:29:01.635358 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:01.635365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:01.635389 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:29:01.635448 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:01.635455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:01.635731 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:01.635897 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:01.636647 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50834: BlockingRecv error: recv got EOF from 127.0.0.1:50834 (error 108)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":310,"threads_started":1}
I20260929 16:29:01.652477 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8830
W20260929 16:29:01.655534 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8830: Not authorized: Client connection negotiation failed: client connection to 127.5.232.131:41389: server requires authentication, but client does not have Kerberos credentials available
I20260929 16:29:01.655736 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 8830
I20260929 16:29:01.671012 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8470
I20260929 16:29:01.672449 8532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:01.779685 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 8470
2026-09-29T16:29:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4823 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:01 dist-test-slave-dssx krb5kdc[8975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:01 dist-test-slave-dssx krb5kdc[8975](info): set up 2 sockets
Sep 29 16:29:01 dist-test-slave-dssx krb5kdc[8975](info): commencing operation
krb5kdc: starting...
W20260929 16:29:02.873816 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:29:02 dist-test-slave-dssx krb5kdc[8975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:29:03.069506 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:43561
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:03.406713 8988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:03.407087 8988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:03.407254 8988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:03.414810 8988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:03.414994 8988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:03.415104 8988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:03.415216 8988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:03.415302 8988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:03.426124 8988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:43561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:03.427696 8988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:03.429173 8988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:03.442353 8994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:03.442435 8996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:03.442374 8993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:03.444021 8988 server_base.cc:1034] running on GCE node
I20260929 16:29:03.445001 8988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:03.446832 8988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:03.448134 8988 hybrid_clock.cc:648] HybridClock initialized: now 1790699343448102 us; error 52 us; skew 500 ppm
I20260929 16:29:03.448575 8988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:03 dist-test-slave-dssx krb5kdc[8975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699343, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:03.452273 8988 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:29:03.454198 8988 webserver.cc:460] Webserver started at http://127.5.232.190:45957/ using document root <none> and password file <none>
I20260929 16:29:03.454981 8988 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:03.455145 8988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:03.455490 8988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:03.458940 8988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2"
format_stamp: "Formatted at 2026-09-29 16:29:03 on dist-test-slave-dssx"
server_key: "8b25ced1cd2fff2f5f2d34ad05353ef9"
server_key_iv: "0d7c28dd92865c60bb9367bc23cf9754"
server_key_version: "encryptionkey@0"
I20260929 16:29:03.459857 8988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2"
format_stamp: "Formatted at 2026-09-29 16:29:03 on dist-test-slave-dssx"
server_key: "8b25ced1cd2fff2f5f2d34ad05353ef9"
server_key_iv: "0d7c28dd92865c60bb9367bc23cf9754"
server_key_version: "encryptionkey@0"
I20260929 16:29:03.465938 8988 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 16:29:03.469828 9003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:03.471405 8988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:03.471622 8988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2"
format_stamp: "Formatted at 2026-09-29 16:29:03 on dist-test-slave-dssx"
server_key: "8b25ced1cd2fff2f5f2d34ad05353ef9"
server_key_iv: "0d7c28dd92865c60bb9367bc23cf9754"
server_key_version: "encryptionkey@0"
I20260929 16:29:03.471856 8988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:03.493927 8988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:03.498492 8988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:03.498855 8988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:03.518636 8988 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:43561
I20260929 16:29:03.518663 9054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:43561 every 8 connection(s)
I20260929 16:29:03.520200 8988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:03.524333 9055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:03.529559 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 8988
I20260929 16:29:03.529829 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:03.530575 6050 external_mini_cluster.cc:1428] Setting key a10fe4fbe705d50575071e872f1f14d3
I20260929 16:29:03.536576 9055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2: Bootstrap starting.
I20260929 16:29:03.540688 9055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:03.542047 9055 log.cc:824] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:03.545055 9055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2: No bootstrap required, opened a new log
Sep 29 16:29:03 dist-test-slave-dssx krb5kdc[8975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:03.553128 9055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43561 } }
I20260929 16:29:03.553746 9055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:03.553895 9055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca1063bdbd047b6bdaab3011e3bd6f2, State: Initialized, Role: FOLLOWER
I20260929 16:29:03.554607 9055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43561 } }
I20260929 16:29:03.554878 9055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:03.555035 9055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:03.555121 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:03.532676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.5.232.190:43561)
0929 16:29:03.533658 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:03.533671 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:03.533721 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:29:03.535479 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:03.535503 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:03.535944 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:03.536311 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:03.538300 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:03.539685 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:03.541582 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:03.542392 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:03.542544 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:03.546864 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:03.546873 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:03.546893 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:03.546945 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:03.550373 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:03.551211 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:03.551228 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:03.551236 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:03.551434 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:03.551899 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:03.551903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:03.551905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:03.552516 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:03.552827 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:03.553328 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 16:29:03.553851 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":450,"threads_started":1}
I20260929 16:29:03.555250 9055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:03.558320 9055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43561 } }
I20260929 16:29:03.558972 9055 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca1063bdbd047b6bdaab3011e3bd6f2; no voters:
I20260929 16:29:03.559693 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:29:03.559686 9055 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:03.559795 9060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:03.561870 9055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:03.569231 9060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [term 1 LEADER]: Becoming Leader. State: Replica: 2ca1063bdbd047b6bdaab3011e3bd6f2, State: Running, Role: LEADER
I20260929 16:29:03.570165 9060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43561 } }
I20260929 16:29:03.573410 9062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca1063bdbd047b6bdaab3011e3bd6f2. Latest consensus state: current_term: 1 leader_uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43561 } } }
I20260929 16:29:03.573807 9062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:03.574532 9061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca1063bdbd047b6bdaab3011e3bd6f2" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43561 } } }
I20260929 16:29:03.574888 9061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:03.582824 9071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:03.590504 9071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:03.607097 9071 catalog_manager.cc:1347] Generated new cluster ID: a1a811bb15c444bebd152b6c5309b1a3
I20260929 16:29:03.607226 9071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:03.621167 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:03.611635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.5.232.190:43561)
0929 16:29:03.611833 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:03.611840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:03.612651 (+ 811us) server_negotiation.cc:407] Connection header received
0929 16:29:03.612872 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:03.612878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:03.612963 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:03.613063 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:03.614398 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:03.615478 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:03.617424 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:03.618075 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:03.618200 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:03.619501 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:03.619507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:03.619513 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:03.620336 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:03.620418 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:03.620445 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:03.620548 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:03.620716 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 16:29:03.620819 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:29:03.626530 9071 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:03.627923 9071 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:03.640210 9071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca1063bdbd047b6bdaab3011e3bd6f2: Generated new TSK 0
I20260929 16:29:03.641106 9071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:29:03.651165 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 8988
I20260929 16:29:03.652465 9050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:03.745698 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 8988
2026-09-29T16:29:03Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1960 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:03 dist-test-slave-dssx krb5kdc[9090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:03 dist-test-slave-dssx krb5kdc[9090](info): set up 2 sockets
Sep 29 16:29:03 dist-test-slave-dssx krb5kdc[9090](info): commencing operation
krb5kdc: starting...
W20260929 16:29:04.817709 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:29:04 dist-test-slave-dssx krb5kdc[9090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:29:05.011098 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:44971
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:05.355607 9103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:05.355932 9103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:05.356086 9103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:05.363979 9103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:05.364109 9103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:05.364207 9103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:05.364317 9103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:05.364403 9103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:05.374930 9103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:05.376366 9103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:05.377780 9103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:05.389993 9108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:05.390062 9111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:05.390024 9109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:05.390928 9103 server_base.cc:1034] running on GCE node
I20260929 16:29:05.391808 9103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:05.393370 9103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:05.394661 9103 hybrid_clock.cc:648] HybridClock initialized: now 1790699345394622 us; error 35 us; skew 500 ppm
I20260929 16:29:05.395057 9103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:05 dist-test-slave-dssx krb5kdc[9090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699345, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:05.403645 9103 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:29:05.405263 9103 webserver.cc:460] Webserver started at http://127.5.232.190:45461/ using document root <none> and password file <none>
I20260929 16:29:05.406042 9103 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:05.406183 9103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:05.406523 9103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:05.409849 9103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "46f3569c71e74d6d9e467cebfb462f8d"
format_stamp: "Formatted at 2026-09-29 16:29:05 on dist-test-slave-dssx"
server_key: "53ca97bf9c57049f5c2e357a5760a620"
server_key_iv: "9de25c9f66aa3f54528b3cc1fd5eca89"
server_key_version: "encryptionkey@0"
I20260929 16:29:05.410707 9103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "46f3569c71e74d6d9e467cebfb462f8d"
format_stamp: "Formatted at 2026-09-29 16:29:05 on dist-test-slave-dssx"
server_key: "53ca97bf9c57049f5c2e357a5760a620"
server_key_iv: "9de25c9f66aa3f54528b3cc1fd5eca89"
server_key_version: "encryptionkey@0"
I20260929 16:29:05.416435 9103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:29:05.420290 9118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:05.421942 9103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:05.422156 9103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "46f3569c71e74d6d9e467cebfb462f8d"
format_stamp: "Formatted at 2026-09-29 16:29:05 on dist-test-slave-dssx"
server_key: "53ca97bf9c57049f5c2e357a5760a620"
server_key_iv: "9de25c9f66aa3f54528b3cc1fd5eca89"
server_key_version: "encryptionkey@0"
I20260929 16:29:05.422360 9103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:05.480686 9103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:05.485038 9103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:05.485402 9103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:05.504938 9103 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:44971
I20260929 16:29:05.504972 9169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:44971 every 8 connection(s)
I20260929 16:29:05.506408 9103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:05.510282 9170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:05.511454 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 9103
I20260929 16:29:05.511687 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:05.512473 6050 external_mini_cluster.cc:1428] Setting key 79e0bd95b67d2eb576041f507d4a8c0a
I20260929 16:29:05.525568 9170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d: Bootstrap starting.
Sep 29 16:29:05 dist-test-slave-dssx krb5kdc[9090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:05.529346 9170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:05.530755 9170 log.cc:824] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:05.534003 9170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d: No bootstrap required, opened a new log
I20260929 16:29:05.536547 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.514743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.5.232.190:44971)
0929 16:29:05.515599 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:05.515615 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:05.515663 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:29:05.517244 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:05.517269 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:05.517795 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:05.518158 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:05.519871 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.521158 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.523070 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.523825 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.523979 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.528805 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:05.528814 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:05.528831 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:05.528880 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:05.532175 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:05.532986 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:05.532993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:05.533000 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:05.533171 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:05.533650 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:05.533657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:05.533660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:05.534255 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:05.534619 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:05.535032 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 16:29:05.535469 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":422,"threads_started":1}
I20260929 16:29:05.542505 9170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f3569c71e74d6d9e467cebfb462f8d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44971 } }
I20260929 16:29:05.542948 9170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:05.543061 9170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f3569c71e74d6d9e467cebfb462f8d, State: Initialized, Role: FOLLOWER
I20260929 16:29:05.543704 9170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f3569c71e74d6d9e467cebfb462f8d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44971 } }
I20260929 16:29:05.543969 9170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:05.544106 9170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:05.544281 9170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:05.546835 9170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f3569c71e74d6d9e467cebfb462f8d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44971 } }
I20260929 16:29:05.547408 9170 leader_election.cc:302] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46f3569c71e74d6d9e467cebfb462f8d; no voters:
I20260929 16:29:05.548043 9170 leader_election.cc:288] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:05.548195 9175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:05.548871 9175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [term 1 LEADER]: Becoming Leader. State: Replica: 46f3569c71e74d6d9e467cebfb462f8d, State: Running, Role: LEADER
I20260929 16:29:05.549516 9175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f3569c71e74d6d9e467cebfb462f8d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44971 } }
I20260929 16:29:05.550266 9170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:05.552382 9176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46f3569c71e74d6d9e467cebfb462f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f3569c71e74d6d9e467cebfb462f8d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44971 } } }
I20260929 16:29:05.552821 9176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:05.552408 9177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46f3569c71e74d6d9e467cebfb462f8d. Latest consensus state: current_term: 1 leader_uuid: "46f3569c71e74d6d9e467cebfb462f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46f3569c71e74d6d9e467cebfb462f8d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44971 } } }
I20260929 16:29:05.556864 9177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:05.557636 9180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:05.562626 9180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:05.571116 9180 catalog_manager.cc:1347] Generated new cluster ID: 90515f56e6104629abd2a37cce07b601
I20260929 16:29:05.571244 9180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:05.602087 9180 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:05.603493 9180 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:05.616609 9180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46f3569c71e74d6d9e467cebfb462f8d: Generated new TSK 0
I20260929 16:29:05.617625 9180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:29:05.669188 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44971
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:06.011708 9196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:06.012053 9196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:06.012228 9196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:06.019548 9196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:06.019729 9196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:06.019919 9196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:06.030156 9196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:06.031513 9196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:06.032863 9196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:06.046211 9201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:06.046249 9202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:06.046227 9204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:06.047173 9196 server_base.cc:1034] running on GCE node
I20260929 16:29:06.047798 9196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:06.048766 9196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:06.049973 9196 hybrid_clock.cc:648] HybridClock initialized: now 1790699346049902 us; error 61 us; skew 500 ppm
I20260929 16:29:06.050344 9196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:06 dist-test-slave-dssx krb5kdc[9090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699346, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:06.053771 9196 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:29:06.055418 9196 webserver.cc:460] Webserver started at http://127.5.232.129:39217/ using document root <none> and password file <none>
I20260929 16:29:06.056239 9196 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:06.056375 9196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:06.056689 9196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:06.059844 9196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "e060363e83d0471d854aa65f60ab9f7b"
format_stamp: "Formatted at 2026-09-29 16:29:06 on dist-test-slave-dssx"
server_key: "52146a0b785642df7437895755288fd6"
server_key_iv: "752547f12027c0fd0128e1b02b2e225a"
server_key_version: "encryptionkey@0"
I20260929 16:29:06.060762 9196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "e060363e83d0471d854aa65f60ab9f7b"
format_stamp: "Formatted at 2026-09-29 16:29:06 on dist-test-slave-dssx"
server_key: "52146a0b785642df7437895755288fd6"
server_key_iv: "752547f12027c0fd0128e1b02b2e225a"
server_key_version: "encryptionkey@0"
I20260929 16:29:06.066656 9196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 16:29:06.070180 9211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:06.071590 9196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:06.071769 9196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "e060363e83d0471d854aa65f60ab9f7b"
format_stamp: "Formatted at 2026-09-29 16:29:06 on dist-test-slave-dssx"
server_key: "52146a0b785642df7437895755288fd6"
server_key_iv: "752547f12027c0fd0128e1b02b2e225a"
server_key_version: "encryptionkey@0"
I20260929 16:29:06.071967 9196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:06.090767 9196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:06.095273 9196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:06.095589 9196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:06.096583 9196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:06.098057 9196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:06.098129 9196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:06.098259 9196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:06.098321 9196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:06.131735 9196 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:40431
I20260929 16:29:06.131769 9323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:40431 every 8 connection(s)
I20260929 16:29:06.133072 9196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:06.143342 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 9196
I20260929 16:29:06.143594 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:06.144258 6050 external_mini_cluster.cc:1428] Setting key 783e4021527c68f55e1da37d7f02a5fc
Sep 29 16:29:06 dist-test-slave-dssx krb5kdc[9090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699346, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:06.154366 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:06.135685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:51719 (local address 127.5.232.190:44971)
0929 16:29:06.136163 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:06.136168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:06.137040 (+ 872us) server_negotiation.cc:407] Connection header received
0929 16:29:06.138805 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:06.138811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:06.138903 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:06.139019 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:06.140576 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:06.141434 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:06.142998 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:06.143403 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:06.143535 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:06.147315 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:06.147324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:06.147333 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:06.147390 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:06.151036 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:06.151822 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:06.151832 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:06.151838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:06.151993 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:06.152514 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:06.152517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:06.152519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:06.152771 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:06.152881 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:06.153855 (+ 974us) server_negotiation.cc:299] Negotiation successful
0929 16:29:06.154030 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":269,"threads_started":1}
I20260929 16:29:06.155949 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:06.136176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44971 (local address 127.5.232.129:51719)
0929 16:29:06.136871 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:06.136914 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:06.138534 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:06.139259 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:06.139268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:06.139832 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:06.140408 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:06.140425 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:06.141593 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:06.141597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:06.142831 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:06.142843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:06.143645 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:06.143652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:06.143897 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:06.144608 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:06.144635 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:06.147090 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:06.151181 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:06.151188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:06.151203 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:06.151706 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:06.152122 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:06.152130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:06.152135 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:06.152398 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:06.152899 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:06.152908 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:06.153458 (+ 550us) client_negotiation.cc:769] Sending connection context
0929 16:29:06.153785 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 16:29:06.154411 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":270,"threads_started":1}
I20260929 16:29:06.157655 9324 heartbeater.cc:347] Connected to a master server at 127.5.232.190:44971
I20260929 16:29:06.157975 9324 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:06.158761 9324 heartbeater.cc:511] Master 127.5.232.190:44971 requested a full tablet report, sending...
I20260929 16:29:06.161055 9135 ts_manager.cc:194] Registered new tserver with Master: e060363e83d0471d854aa65f60ab9f7b (127.5.232.129:40431)
I20260929 16:29:06.162619 9135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:51719
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:29:06.188529 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44971
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:06.518433 9330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:06.518719 9330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:06.518813 9330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:06.526743 9330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:06.526881 9330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:06.527062 9330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:06.537498 9330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:06.538885 9330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:06.540292 9330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:06.554037 9338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:06.554160 9335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:06.554134 9336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:06.555115 9330 server_base.cc:1034] running on GCE node
I20260929 16:29:06.555939 9330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:06.556978 9330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:06.558238 9330 hybrid_clock.cc:648] HybridClock initialized: now 1790699346558181 us; error 50 us; skew 500 ppm
I20260929 16:29:06.558614 9330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:06 dist-test-slave-dssx krb5kdc[9090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699346, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:06.561981 9330 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:29:06.563534 9330 webserver.cc:460] Webserver started at http://127.5.232.130:42601/ using document root <none> and password file <none>
I20260929 16:29:06.564272 9330 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:06.564376 9330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:06.564635 9330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:06.567621 9330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "4e997f017dd5467ebd864d77ded732ff"
format_stamp: "Formatted at 2026-09-29 16:29:06 on dist-test-slave-dssx"
server_key: "83f70fd3aff6637a44f5bcf26bee4083"
server_key_iv: "af46eec58f0d781ac485bb95def8561b"
server_key_version: "encryptionkey@0"
I20260929 16:29:06.568372 9330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "4e997f017dd5467ebd864d77ded732ff"
format_stamp: "Formatted at 2026-09-29 16:29:06 on dist-test-slave-dssx"
server_key: "83f70fd3aff6637a44f5bcf26bee4083"
server_key_iv: "af46eec58f0d781ac485bb95def8561b"
server_key_version: "encryptionkey@0"
I20260929 16:29:06.573843 9330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:29:06.577605 9345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:06.579015 9330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:06.579180 9330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "4e997f017dd5467ebd864d77ded732ff"
format_stamp: "Formatted at 2026-09-29 16:29:06 on dist-test-slave-dssx"
server_key: "83f70fd3aff6637a44f5bcf26bee4083"
server_key_iv: "af46eec58f0d781ac485bb95def8561b"
server_key_version: "encryptionkey@0"
I20260929 16:29:06.579382 9330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:06.603675 9330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:06.608210 9330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:06.608525 9330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:06.609558 9330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:06.610884 9330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:06.610958 9330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:06.611085 9330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:06.611141 9330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:06.644866 9330 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45587
I20260929 16:29:06.644925 9457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45587 every 8 connection(s)
I20260929 16:29:06.646310 9330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:06.647845 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 9330
I20260929 16:29:06.648063 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:06.648578 6050 external_mini_cluster.cc:1428] Setting key a9dd25f985dc49506edf96d841c46aa9
Sep 29 16:29:06 dist-test-slave-dssx krb5kdc[9090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699346, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
I20260929 16:29:06.670365 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:06.649119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:40419 (local address 127.5.232.190:44971)
0929 16:29:06.649331 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:06.649337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:06.650905 (+ 1568us) server_negotiation.cc:407] Connection header received
0929 16:29:06.653923 (+ 3018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:06.653927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:06.654002 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:06.654096 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:06.656545 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:06.657467 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:06.659546 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:06.659981 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:06.660129 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:06.664337 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:06.664343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:06.664346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:06.664381 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:06.667058 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:06.667859 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:06.667865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:06.667869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:06.668005 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:06.668452 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:06.668459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:06.668462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:06.668798 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:06.668958 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:06.669778 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 16:29:06.670016 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
I20260929 16:29:06.671590 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:06.649827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44971 (local address 127.5.232.130:40419)
0929 16:29:06.650740 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:06.650791 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:06.653565 (+ 2774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:06.654471 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:06.654499 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:06.655405 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:06.656321 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:06.656353 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:06.657641 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:06.657648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:06.659369 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:06.659383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:06.660228 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:06.660234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:06.660491 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:06.661354 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:06.661451 (+ 97us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:06.664052 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:06.667231 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:06.667241 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:06.667262 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:06.667726 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:06.668125 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:06.668131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:06.668137 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:06.668336 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:06.668940 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:06.668949 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:06.669324 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 16:29:06.669704 (+ 380us) client_negotiation.cc:240] Negotiation successful
0929 16:29:06.670317 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":305,"threads_started":1}
I20260929 16:29:06.673246 9458 heartbeater.cc:347] Connected to a master server at 127.5.232.190:44971
I20260929 16:29:06.673600 9458 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:06.674376 9458 heartbeater.cc:511] Master 127.5.232.190:44971 requested a full tablet report, sending...
I20260929 16:29:06.676132 9135 ts_manager.cc:194] Registered new tserver with Master: 4e997f017dd5467ebd864d77ded732ff (127.5.232.130:45587)
I20260929 16:29:06.677287 9135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:40419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:29:06.694725 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44971
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:07.040663 9463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:07.041024 9463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:07.041205 9463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:07.048820 9463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:07.048995 9463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:07.049181 9463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:07.059937 9463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:07.061576 9463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:07.063093 9463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:07.076166 9471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:07.076090 9469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:07.076179 9468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:07.076948 9463 server_base.cc:1034] running on GCE node
I20260929 16:29:07.077666 9463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:07.078704 9463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:07.079910 9463 hybrid_clock.cc:648] HybridClock initialized: now 1790699347079852 us; error 50 us; skew 500 ppm
I20260929 16:29:07.080256 9463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699347, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:07.084054 9463 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:29:07.085666 9463 webserver.cc:460] Webserver started at http://127.5.232.131:34123/ using document root <none> and password file <none>
I20260929 16:29:07.086395 9463 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:07.086508 9463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:07.086764 9463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:07.089850 9463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "c120d9cf5e284ee6828807b5b05197ad"
format_stamp: "Formatted at 2026-09-29 16:29:07 on dist-test-slave-dssx"
server_key: "0df365762153ff7b7d3451629d9aa11d"
server_key_iv: "2e0913bc0a7a192194c81ae07d5134e1"
server_key_version: "encryptionkey@0"
I20260929 16:29:07.090612 9463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "c120d9cf5e284ee6828807b5b05197ad"
format_stamp: "Formatted at 2026-09-29 16:29:07 on dist-test-slave-dssx"
server_key: "0df365762153ff7b7d3451629d9aa11d"
server_key_iv: "2e0913bc0a7a192194c81ae07d5134e1"
server_key_version: "encryptionkey@0"
I20260929 16:29:07.096202 9463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:29:07.099928 9478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:07.101352 9463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:07.101573 9463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "c120d9cf5e284ee6828807b5b05197ad"
format_stamp: "Formatted at 2026-09-29 16:29:07 on dist-test-slave-dssx"
server_key: "0df365762153ff7b7d3451629d9aa11d"
server_key_iv: "2e0913bc0a7a192194c81ae07d5134e1"
server_key_version: "encryptionkey@0"
I20260929 16:29:07.101759 9463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:07.116734 9463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:07.121577 9463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:07.121884 9463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:07.122905 9463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:07.124264 9463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:07.124387 9463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:07.124519 9463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:07.124574 9463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:07.158196 9463 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:34105
I20260929 16:29:07.158231 9590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:34105 every 8 connection(s)
I20260929 16:29:07.159529 9463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:07.165591 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 9463
I20260929 16:29:07.165807 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:07.166119 9324 heartbeater.cc:503] Master 127.5.232.190:44971 was elected leader, sending a full tablet report...
I20260929 16:29:07.166374 6050 external_mini_cluster.cc:1428] Setting key 27d94f5c0b79d551571e7b48b7b08b37
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699347, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:07.180528 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:07.162349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:49765 (local address 127.5.232.190:44971)
0929 16:29:07.162572 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:07.162578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:07.163766 (+ 1188us) server_negotiation.cc:407] Connection header received
0929 16:29:07.165764 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:07.165769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:07.165899 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:07.166007 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:07.168030 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.168829 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.170588 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.171008 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.171150 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:07.174662 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:07.174667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:07.174670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:07.174701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:07.177447 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.178205 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.178210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.178213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.178340 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.178769 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.178773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.178777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.179040 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:07.179197 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:07.179932 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 16:29:07.180157 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 16:29:07.181788 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:07.162886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44971 (local address 127.5.232.131:49765)
0929 16:29:07.163597 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:07.163635 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:07.165360 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:07.166274 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:07.166286 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:07.167031 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:07.167870 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:07.167891 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.168980 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.168985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:07.170441 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:07.170452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.171253 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.171260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:07.171501 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:07.172145 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:07.172168 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:07.174432 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:07.177609 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:07.177619 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:07.177637 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:07.178080 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:07.178450 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:07.178455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:07.178459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:07.178653 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:07.179171 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:07.179180 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:07.179557 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 16:29:07.179827 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 16:29:07.180378 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":273,"threads_started":1}
I20260929 16:29:07.183136 9591 heartbeater.cc:347] Connected to a master server at 127.5.232.190:44971
I20260929 16:29:07.183461 9591 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:07.184208 9591 heartbeater.cc:511] Master 127.5.232.190:44971 requested a full tablet report, sending...
I20260929 16:29:07.185570 9135 ts_manager.cc:194] Registered new tserver with Master: c120d9cf5e284ee6828807b5b05197ad (127.5.232.131:34105)
I20260929 16:29:07.186368 9135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:49765
I20260929 16:29:07.194319 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:07.245081 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:07.231045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.5.232.190:44971)
0929 16:29:07.231250 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:07.231254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:07.231756 (+ 502us) server_negotiation.cc:407] Connection header received
0929 16:29:07.231910 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:07.231914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:07.231986 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:07.232084 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:07.233573 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.234320 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.235565 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.235976 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.236071 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:07.240004 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:07.240009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:07.240013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:07.240049 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:07.242851 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.243514 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.243517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.243520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.243647 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.243960 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.243963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.243965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.244285 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:07.244482 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:07.244705 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 16:29:07.244862 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 16:29:07.245675 9135 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46106
I20260929 16:29:07.247118 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9196
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:29:07.263908 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:07.248339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.5.232.129:40431)
0929 16:29:07.248970 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:07.248978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:07.249003 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:29:07.249062 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:07.249068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:07.249287 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:07.249475 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:07.250669 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.252035 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.253229 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.253862 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.253969 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:07.258115 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:07.258128 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:07.258144 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:07.258180 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:07.261051 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.261904 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.261911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.261914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.262043 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.262379 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.262382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.262384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.262790 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:07.263061 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:07.263314 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:29:07.263510 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":334,"threads_started":1}
W20260929 16:29:07.265197 9319 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53026
W20260929 16:29:07.266424 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9196: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:29:07.266633 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 9196
I20260929 16:29:07.281877 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9330
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:29:07.297780 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:07.282817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.5.232.130:45587)
0929 16:29:07.283365 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:07.283372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:07.283397 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:29:07.283478 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:07.283484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:07.283756 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:07.283903 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:07.285129 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.286212 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.287371 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.287884 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.287981 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:07.292042 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:07.292053 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:07.292068 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:07.292101 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:07.295025 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.295701 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.295708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.295711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.295825 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.296195 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.296202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.296205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.296531 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:07.296854 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:07.297150 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 16:29:07.297447 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":304,"threads_started":1}
W20260929 16:29:07.299046 9453 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54536
W20260929 16:29:07.300109 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9330: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:29:07.300280 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 9330
I20260929 16:29:07.315722 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9463
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699347, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:29:07.331406 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:07.316641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.5.232.131:34105)
0929 16:29:07.317274 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:07.317281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:07.317310 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:29:07.317367 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:07.317425 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:07.317665 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:07.317790 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:07.318973 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.319993 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.321100 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:07.321793 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:07.321892 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:07.326096 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:07.326107 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:07.326121 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:07.326154 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:07.329010 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.329666 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.329673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.329676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.329785 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:07.330073 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:07.330076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:07.330078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:07.330385 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:07.330605 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:07.330812 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 16:29:07.331003 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":343,"threads_started":1}
W20260929 16:29:07.332712 9586 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60196
W20260929 16:29:07.333860 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9463: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:29:07.334046 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 9463
I20260929 16:29:07.349303 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9103
I20260929 16:29:07.350600 9165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:07.451272 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 9103
2026-09-29T16:29:07Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3711 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9608](info): set up 2 sockets
Sep 29 16:29:07 dist-test-slave-dssx krb5kdc[9608](info): commencing operation
krb5kdc: starting...
W20260929 16:29:08.529824 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:29:08 dist-test-slave-dssx krb5kdc[9608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:29:08.720983 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40643
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:09.049842 9621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:09.050158 9621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:09.050247 9621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:09.057686 9621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:09.057798 9621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:09.057847 9621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:09.057904 9621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:09.057948 9621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:09.068027 9621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:40643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:09.069324 9621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:09.070669 9621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:09.082577 9626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:09.082463 9627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:09.082674 9629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:09.083521 9621 server_base.cc:1034] running on GCE node
I20260929 16:29:09.084434 9621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:09.086035 9621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:09.087276 9621 hybrid_clock.cc:648] HybridClock initialized: now 1790699349087215 us; error 54 us; skew 500 ppm
I20260929 16:29:09.087656 9621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:09 dist-test-slave-dssx krb5kdc[9608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699349, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:09.090845 9621 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:29:09.092344 9621 webserver.cc:460] Webserver started at http://127.5.232.190:34007/ using document root <none> and password file <none>
I20260929 16:29:09.092994 9621 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:09.093108 9621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:09.093432 9621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:09.096415 9621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "722f7c3005f94f9794d02d5d6db30c94"
format_stamp: "Formatted at 2026-09-29 16:29:09 on dist-test-slave-dssx"
server_key: "558d49c65dc46f5d0762bf76925295a8"
server_key_iv: "7e8b622b74f80a853ccf2c72b8636311"
server_key_version: "encryptionkey@0"
I20260929 16:29:09.097229 9621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "722f7c3005f94f9794d02d5d6db30c94"
format_stamp: "Formatted at 2026-09-29 16:29:09 on dist-test-slave-dssx"
server_key: "558d49c65dc46f5d0762bf76925295a8"
server_key_iv: "7e8b622b74f80a853ccf2c72b8636311"
server_key_version: "encryptionkey@0"
I20260929 16:29:09.102412 9621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:29:09.105824 9636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:09.107203 9621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:29:09.107336 9621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "722f7c3005f94f9794d02d5d6db30c94"
format_stamp: "Formatted at 2026-09-29 16:29:09 on dist-test-slave-dssx"
server_key: "558d49c65dc46f5d0762bf76925295a8"
server_key_iv: "7e8b622b74f80a853ccf2c72b8636311"
server_key_version: "encryptionkey@0"
I20260929 16:29:09.107494 9621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:09.124779 9621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:09.128824 9621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:09.129062 9621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:09.148038 9621 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:40643
I20260929 16:29:09.148059 9687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:40643 every 8 connection(s)
I20260929 16:29:09.149600 9621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:09.153749 9688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:09.159545 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 9621
I20260929 16:29:09.159777 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:09.160588 6050 external_mini_cluster.cc:1428] Setting key 7fa763ec77ee45772d48955cb878bf82
I20260929 16:29:09.165722 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94: Bootstrap starting.
I20260929 16:29:09.169675 9688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:09.170989 9688 log.cc:824] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:09.173955 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94: No bootstrap required, opened a new log
Sep 29 16:29:09 dist-test-slave-dssx krb5kdc[9608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:09.182139 9688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722f7c3005f94f9794d02d5d6db30c94" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40643 } }
I20260929 16:29:09.182591 9688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:09.182723 9688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722f7c3005f94f9794d02d5d6db30c94, State: Initialized, Role: FOLLOWER
I20260929 16:29:09.183493 9688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722f7c3005f94f9794d02d5d6db30c94" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40643 } }
I20260929 16:29:09.183758 9688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:09.183911 9688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:09.184101 9688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:09.185230 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:09.162901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46676 (local address 127.5.232.190:40643)
0929 16:29:09.163818 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:09.163831 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:09.163874 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:29:09.165814 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:09.165850 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:09.166301 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:09.166660 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:09.168561 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:09.170025 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:09.171913 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:09.172630 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:09.172833 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:09.177102 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:09.177111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:09.177132 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:09.177178 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:09.180562 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:09.181349 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:09.181358 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:09.181365 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:09.181613 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:09.182001 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:09.182007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:09.182010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:09.182712 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:09.183037 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:09.183501 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 16:29:09.183994 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":454,"threads_started":1}
I20260929 16:29:09.187083 9688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722f7c3005f94f9794d02d5d6db30c94" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40643 } }
I20260929 16:29:09.187702 9688 leader_election.cc:302] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 722f7c3005f94f9794d02d5d6db30c94; no voters:
I20260929 16:29:09.188342 9688 leader_election.cc:288] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:09.188490 9693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:09.189239 9693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [term 1 LEADER]: Becoming Leader. State: Replica: 722f7c3005f94f9794d02d5d6db30c94, State: Running, Role: LEADER
I20260929 16:29:09.189829 9693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722f7c3005f94f9794d02d5d6db30c94" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40643 } }
I20260929 16:29:09.190604 9688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:09.192919 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "722f7c3005f94f9794d02d5d6db30c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722f7c3005f94f9794d02d5d6db30c94" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40643 } } }
I20260929 16:29:09.193213 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:09.193609 9695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722f7c3005f94f9794d02d5d6db30c94. Latest consensus state: current_term: 1 leader_uuid: "722f7c3005f94f9794d02d5d6db30c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722f7c3005f94f9794d02d5d6db30c94" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40643 } } }
I20260929 16:29:09.193956 9695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:09.194593 9700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:09.200848 9700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:09.214803 9700 catalog_manager.cc:1347] Generated new cluster ID: be2f71659605477da640d4db10686bf1
I20260929 16:29:09.214929 9700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:09.284747 9700 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:09.286705 9700 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:09.298096 9700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 722f7c3005f94f9794d02d5d6db30c94: Generated new TSK 0
I20260929 16:29:09.299177 9700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:29:09.352902 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40643
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:09.684037 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:09.684388 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:09.684556 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:09.692222 9714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:09.692384 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:09.692555 9714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:09.702891 9714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:09.704293 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:09.705761 9714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:09.719600 9719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:09.719677 9720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:09.719637 9722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:09.720925 9714 server_base.cc:1034] running on GCE node
I20260929 16:29:09.721669 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:09.722805 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:09.724078 9714 hybrid_clock.cc:648] HybridClock initialized: now 1790699349724031 us; error 81 us; skew 500 ppm
I20260929 16:29:09.724387 9714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:09 dist-test-slave-dssx krb5kdc[9608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699349, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:09.728492 9714 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:29:09.730110 9714 webserver.cc:460] Webserver started at http://127.5.232.129:43105/ using document root <none> and password file <none>
I20260929 16:29:09.730844 9714 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:09.730980 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:09.731317 9714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:09.734408 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "4dcb42ec7ef74c558b9406140d1223f8"
format_stamp: "Formatted at 2026-09-29 16:29:09 on dist-test-slave-dssx"
server_key: "9c74fade46acdf8fabc539dd31ad14e4"
server_key_iv: "95edf7fc54236c7603c9f6e4d38f9644"
server_key_version: "encryptionkey@0"
I20260929 16:29:09.735196 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "4dcb42ec7ef74c558b9406140d1223f8"
format_stamp: "Formatted at 2026-09-29 16:29:09 on dist-test-slave-dssx"
server_key: "9c74fade46acdf8fabc539dd31ad14e4"
server_key_iv: "95edf7fc54236c7603c9f6e4d38f9644"
server_key_version: "encryptionkey@0"
I20260929 16:29:09.740466 9714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:29:09.744237 9729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:09.745790 9714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:09.745983 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "4dcb42ec7ef74c558b9406140d1223f8"
format_stamp: "Formatted at 2026-09-29 16:29:09 on dist-test-slave-dssx"
server_key: "9c74fade46acdf8fabc539dd31ad14e4"
server_key_iv: "95edf7fc54236c7603c9f6e4d38f9644"
server_key_version: "encryptionkey@0"
I20260929 16:29:09.746203 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:09.767478 9714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:09.771696 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:09.772001 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:09.772976 9714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:09.774410 9714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:09.774487 9714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:09.774574 9714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:09.774618 9714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:09.807900 9714 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:37791
I20260929 16:29:09.807948 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:37791 every 8 connection(s)
I20260929 16:29:09.809206 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:09.814138 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 9714
I20260929 16:29:09.814347 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:09.814968 6050 external_mini_cluster.cc:1428] Setting key b65ed0f46c86f5a581ef13f71b873ece
Sep 29 16:29:09 dist-test-slave-dssx krb5kdc[9608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699349, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:09.829701 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:09.811930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:59695 (local address 127.5.232.190:40643)
0929 16:29:09.812351 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:09.812355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:09.813452 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 16:29:09.815460 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:09.815466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:09.815557 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:09.815676 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:09.817471 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:09.818205 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:09.819788 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:09.820172 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:09.820311 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:09.824027 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:09.824032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:09.824035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:09.824067 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:09.826589 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:09.827384 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:09.827389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:09.827391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:09.827505 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:09.827941 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:09.827946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:09.827949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:09.828212 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:09.828365 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:09.829083 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 16:29:09.829302 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":210,"threads_started":1}
I20260929 16:29:09.830714 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:09.812568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40643 (local address 127.5.232.129:59695)
0929 16:29:09.813267 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:09.813310 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:09.815179 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:09.815947 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:09.815960 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:09.816692 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:09.817276 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:09.817293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:09.818361 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:09.818366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:09.819619 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:09.819630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:09.820411 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:09.820420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:09.820654 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:09.821342 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:09.821370 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:09.823787 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:09.826759 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:09.826769 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:09.826789 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:09.827231 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:09.827619 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:09.827624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:09.827627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:09.827823 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:09.828338 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:09.828347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:09.828713 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 16:29:09.828985 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 16:29:09.829584 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":266,"threads_started":1}
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
I20260929 16:29:09.832458 9842 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40643
I20260929 16:29:09.832764 9842 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:09.833555 9842 heartbeater.cc:511] Master 127.5.232.190:40643 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
I20260929 16:29:09.835693 9653 ts_manager.cc:194] Registered new tserver with Master: 4dcb42ec7ef74c558b9406140d1223f8 (127.5.232.129:37791)
I20260929 16:29:09.837114 9653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:59695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:29:09.857172 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40643
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:10.185549 9848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:10.185856 9848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:10.186039 9848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:10.193766 9848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:10.193924 9848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:10.194121 9848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:10.204492 9848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:10.205957 9848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:10.207376 9848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:10.221295 9853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:10.221995 9856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:10.222167 9854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:10.222688 9848 server_base.cc:1034] running on GCE node
I20260929 16:29:10.223567 9848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:10.224647 9848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:10.225860 9848 hybrid_clock.cc:648] HybridClock initialized: now 1790699350225811 us; error 34 us; skew 500 ppm
I20260929 16:29:10.226200 9848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:10 dist-test-slave-dssx krb5kdc[9608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699350, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:10.229488 9848 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:29:10.230985 9848 webserver.cc:460] Webserver started at http://127.5.232.130:37477/ using document root <none> and password file <none>
I20260929 16:29:10.231684 9848 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:10.231792 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:10.232053 9848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:10.235179 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "e05eece981904d4f9452bbe2eeefd293"
format_stamp: "Formatted at 2026-09-29 16:29:10 on dist-test-slave-dssx"
server_key: "c9e8acb29d5c0179ab11b70a128f4e8b"
server_key_iv: "44fa6c9be13b3963efc0bf88232252f0"
server_key_version: "encryptionkey@0"
I20260929 16:29:10.236081 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "e05eece981904d4f9452bbe2eeefd293"
format_stamp: "Formatted at 2026-09-29 16:29:10 on dist-test-slave-dssx"
server_key: "c9e8acb29d5c0179ab11b70a128f4e8b"
server_key_iv: "44fa6c9be13b3963efc0bf88232252f0"
server_key_version: "encryptionkey@0"
I20260929 16:29:10.241654 9848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:29:10.245496 9863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:10.247067 9848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:10.247283 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "e05eece981904d4f9452bbe2eeefd293"
format_stamp: "Formatted at 2026-09-29 16:29:10 on dist-test-slave-dssx"
server_key: "c9e8acb29d5c0179ab11b70a128f4e8b"
server_key_iv: "44fa6c9be13b3963efc0bf88232252f0"
server_key_version: "encryptionkey@0"
I20260929 16:29:10.247524 9848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:10.273367 9848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:10.277776 9848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:10.278098 9848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:10.279135 9848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:10.280593 9848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:10.280674 9848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:10.280830 9848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:10.280885 9848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:10.315377 9848 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:35481
I20260929 16:29:10.315428 9975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:35481 every 8 connection(s)
I20260929 16:29:10.316745 9848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:10.317449 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 9848
I20260929 16:29:10.317623 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:10.318217 6050 external_mini_cluster.cc:1428] Setting key e3c28698b7762b53813b9d2038a564a1
Sep 29 16:29:10 dist-test-slave-dssx krb5kdc[9608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699350, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
I20260929 16:29:10.339622 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:10.319952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:51795 (local address 127.5.232.190:40643)
0929 16:29:10.320151 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:10.320158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:10.321749 (+ 1591us) server_negotiation.cc:407] Connection header received
0929 16:29:10.323690 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:10.323695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:10.323791 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:10.323920 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:10.325638 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.326359 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.328023 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.328461 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.328612 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:10.332557 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:10.332563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:10.332566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:10.332605 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:10.335674 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.336612 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.336618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.336621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.336814 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.337316 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.337322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.337326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.337832 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:10.338018 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:10.338979 (+ 961us) server_negotiation.cc:299] Negotiation successful
0929 16:29:10.339251 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:29:10.340719 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:10.320712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40643 (local address 127.5.232.130:51795)
0929 16:29:10.321585 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:10.321631 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:10.323365 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:10.324191 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:10.324201 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:10.324759 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:10.325430 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:10.325461 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.326547 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.326553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:10.327846 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:10.327864 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.328717 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.328724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:10.328968 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:10.329756 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:10.329780 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:10.332275 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:10.335893 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:10.335904 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:10.335935 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:10.336434 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:10.336946 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:10.336952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:10.336956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:10.337174 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:10.337996 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:10.338005 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:10.338480 (+ 475us) client_negotiation.cc:769] Sending connection context
0929 16:29:10.338809 (+ 329us) client_negotiation.cc:240] Negotiation successful
0929 16:29:10.339426 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":340,"threads_started":1}
I20260929 16:29:10.342378 9976 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40643
I20260929 16:29:10.342727 9976 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:10.343538 9976 heartbeater.cc:511] Master 127.5.232.190:40643 requested a full tablet report, sending...
I20260929 16:29:10.345135 9653 ts_manager.cc:194] Registered new tserver with Master: e05eece981904d4f9452bbe2eeefd293 (127.5.232.130:35481)
I20260929 16:29:10.346115 9653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:51795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:29:10.364689 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40643
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:10.710281 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:10.710656 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:10.710758 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:10.718641 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:10.718773 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:10.718890 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:10.729290 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:10.730640 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:10.731895 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:10.745139 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:10.745198 9989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:10.745221 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:10.746732 9981 server_base.cc:1034] running on GCE node
I20260929 16:29:10.747440 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:10.748453 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:10.749661 9981 hybrid_clock.cc:648] HybridClock initialized: now 1790699350749615 us; error 40 us; skew 500 ppm
I20260929 16:29:10.750005 9981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:10 dist-test-slave-dssx krb5kdc[9608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699350, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:10.756028 9981 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:29:10.757704 9981 webserver.cc:460] Webserver started at http://127.5.232.131:37981/ using document root <none> and password file <none>
I20260929 16:29:10.758462 9981 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:10.758564 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:10.758821 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:10.761819 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "a5488f1da04744b98777108cb911f0d9"
format_stamp: "Formatted at 2026-09-29 16:29:10 on dist-test-slave-dssx"
server_key: "6609652cc97ad1e2824aca148d64b75c"
server_key_iv: "23f1043a5f9597b61a2b1b7b52e585e1"
server_key_version: "encryptionkey@0"
I20260929 16:29:10.762593 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "a5488f1da04744b98777108cb911f0d9"
format_stamp: "Formatted at 2026-09-29 16:29:10 on dist-test-slave-dssx"
server_key: "6609652cc97ad1e2824aca148d64b75c"
server_key_iv: "23f1043a5f9597b61a2b1b7b52e585e1"
server_key_version: "encryptionkey@0"
I20260929 16:29:10.767946 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:29:10.771760 9996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:10.773146 9981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:10.773339 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "a5488f1da04744b98777108cb911f0d9"
format_stamp: "Formatted at 2026-09-29 16:29:10 on dist-test-slave-dssx"
server_key: "6609652cc97ad1e2824aca148d64b75c"
server_key_iv: "23f1043a5f9597b61a2b1b7b52e585e1"
server_key_version: "encryptionkey@0"
I20260929 16:29:10.773634 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:10.807113 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:10.812004 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:10.812330 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:10.813328 9981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:10.814669 9981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:10.814785 9981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:10.814935 9981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:10.815008 9981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:10.840544 9842 heartbeater.cc:503] Master 127.5.232.190:40643 was elected leader, sending a full tablet report...
I20260929 16:29:10.848560 9981 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:33037
I20260929 16:29:10.848618 10108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:33037 every 8 connection(s)
I20260929 16:29:10.850008 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:10.856287 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 9981
I20260929 16:29:10.856475 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:10.857028 6050 external_mini_cluster.cc:1428] Setting key 4c234f06e350fbc8a860e03ea74e9d76
Sep 29 16:29:10 dist-test-slave-dssx krb5kdc[9608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699350, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:10.872042 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:10.852815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:47545 (local address 127.5.232.190:40643)
0929 16:29:10.853276 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:10.853283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:10.854383 (+ 1100us) server_negotiation.cc:407] Connection header received
0929 16:29:10.856433 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:10.856440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:10.856557 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:10.856669 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:10.858865 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.859981 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.861854 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.862250 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.862347 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:10.866077 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:10.866082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:10.866086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:10.866121 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:10.868602 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.869365 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.869369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.869371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.869572 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.870005 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.870008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.870010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.870275 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:10.870389 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:10.871458 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0929 16:29:10.871696 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":200,"threads_started":1}
I20260929 16:29:10.873181 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:10.853506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40643 (local address 127.5.232.131:47545)
0929 16:29:10.854209 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:10.854253 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:10.856043 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:10.856948 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:10.856960 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:10.857760 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:10.858667 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:10.858704 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.860161 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.860167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:10.861721 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:10.861734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.862436 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.862440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:10.862651 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:10.863407 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:10.863431 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:10.865787 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:10.868765 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:10.868772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:10.868788 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:10.869237 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:10.869701 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:10.869705 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:10.869707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:10.869898 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:10.870428 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:10.870435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:10.870901 (+ 466us) client_negotiation.cc:769] Sending connection context
0929 16:29:10.871271 (+ 370us) client_negotiation.cc:240] Negotiation successful
0929 16:29:10.871869 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":271,"threads_started":1}
I20260929 16:29:10.874485 10109 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40643
I20260929 16:29:10.874797 10109 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:10.875511 10109 heartbeater.cc:511] Master 127.5.232.190:40643 requested a full tablet report, sending...
I20260929 16:29:10.876869 9653 ts_manager.cc:194] Registered new tserver with Master: a5488f1da04744b98777108cb911f0d9 (127.5.232.131:33037)
I20260929 16:29:10.877764 9653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:47545
I20260929 16:29:10.885121 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:29:10 dist-test-slave-dssx krb5kdc[9608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:29:10 dist-test-slave-dssx krb5kdc[9608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:29:10.933225 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:10.919042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.5.232.129:37791)
0929 16:29:10.919811 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:10.919818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:10.919847 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:29:10.919904 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:10.919911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:10.920126 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:10.920255 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:10.921666 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.922673 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.923880 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.924432 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.924527 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:10.928262 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:10.928274 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:10.928291 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:10.928328 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:10.930926 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.931546 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.931552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.931555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.931665 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.931973 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.931976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.931978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.932297 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:10.932520 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:10.932751 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:29:10.932943 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":463,"threads_started":1}
Sep 29 16:29:10 dist-test-slave-dssx krb5kdc[9608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:10.950904 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:10.937478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.5.232.190:40643)
0929 16:29:10.937712 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:10.937718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:10.937892 (+ 174us) server_negotiation.cc:407] Connection header received
0929 16:29:10.938020 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:10.938024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:10.938101 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:10.938183 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:10.939658 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.940358 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.941608 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.942001 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.942093 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:10.946044 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:10.946049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:10.946053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:10.946087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:10.948404 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.948998 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.949002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.949004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.949124 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.949514 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.949518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.949520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.949891 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:10.950075 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:10.950374 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 16:29:10.950541 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 16:29:10.951454 9653 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46690
I20260929 16:29:10.952512 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9714
I20260929 16:29:10.964708 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:10.953525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.5.232.129:37791)
0929 16:29:10.953686 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:10.953690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:10.953923 (+ 233us) server_negotiation.cc:407] Connection header received
0929 16:29:10.954040 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:10.954043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:10.954127 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:10.954210 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:10.955359 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.956027 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.957206 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:10.957681 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:10.957813 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:10.959936 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:10.959941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:10.959945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:10.959976 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:10.962599 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.963206 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.963210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.963212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.963337 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:10.963654 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:10.963657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:10.963659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:10.963897 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:10.964065 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:10.964321 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 16:29:10.964479 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:29:10.965317 9837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:11.049125 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 9714
I20260929 16:29:11.066149 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9848
Sep 29 16:29:11 dist-test-slave-dssx krb5kdc[9608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:29:11.083205 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:11.067048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.5.232.130:35481)
0929 16:29:11.067626 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:11.067634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:11.067660 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:29:11.067721 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:11.067728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:11.067950 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:11.068086 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:11.069263 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:11.070322 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:11.071759 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:11.072352 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:11.072515 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:11.076828 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:11.076843 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:11.076860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:11.076901 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:11.079935 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:11.080927 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:11.080936 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:11.080938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:11.081070 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:11.081524 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:11.081531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:11.081539 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:11.081904 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:11.082237 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:11.082551 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:29:11.082825 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":317,"threads_started":1}
I20260929 16:29:11.084645 9971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:11.168985 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 9848
I20260929 16:29:11.185623 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9981
Sep 29 16:29:11 dist-test-slave-dssx krb5kdc[9608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:29:11.202431 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:11.186643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.5.232.131:33037)
0929 16:29:11.187268 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:11.187275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:11.187305 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:29:11.187363 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:11.187370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:11.187585 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:11.187760 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:11.189036 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:11.190138 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:11.191570 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:11.192187 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:11.192338 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:11.196689 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:11.196702 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:11.196718 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:11.196761 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:11.199523 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:11.200286 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:11.200294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:11.200297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:11.200431 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:11.200807 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:11.200814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:11.200817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:11.201156 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:11.201559 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:11.201792 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 16:29:11.202074 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":324,"threads_started":1}
I20260929 16:29:11.203794 10104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:11.287604 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 9981
I20260929 16:29:11.303591 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 9621
I20260929 16:29:11.304844 9683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:11.406863 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 9621
2026-09-29T16:29:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3955 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:11 dist-test-slave-dssx krb5kdc[10129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:11 dist-test-slave-dssx krb5kdc[10129](info): set up 2 sockets
Sep 29 16:29:11 dist-test-slave-dssx krb5kdc[10129](info): commencing operation
krb5kdc: starting...
W20260929 16:29:12.502300 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:29:12 dist-test-slave-dssx krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:29:12.703255 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:35201
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:35201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:13.035864 10142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:13.036264 10142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:13.036443 10142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:13.043829 10142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:13.044021 10142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:13.044140 10142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:13.044250 10142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:13.044353 10142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:13.054559 10142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:35201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:35201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:13.055959 10142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:13.057335 10142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:13.068869 10148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:13.068827 10150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:13.068920 10147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:13.069855 10142 server_base.cc:1034] running on GCE node
I20260929 16:29:13.070647 10142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:13.072093 10142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:13.073329 10142 hybrid_clock.cc:648] HybridClock initialized: now 1790699353073285 us; error 53 us; skew 500 ppm
I20260929 16:29:13.073714 10142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:13 dist-test-slave-dssx krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:13.076889 10142 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:29:13.078181 10142 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:13.078337 10142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:13.078673 10142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:13.081897 10142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "063f664529a9454198671b61f262e224"
format_stamp: "Formatted at 2026-09-29 16:29:13 on dist-test-slave-dssx"
server_key: "3841b75a6bc07b546937e2d0a8d52ba8"
server_key_iv: "ac936998cf33d29c717a8df4cc5116e1"
server_key_version: "encryptionkey@0"
I20260929 16:29:13.082753 10142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "063f664529a9454198671b61f262e224"
format_stamp: "Formatted at 2026-09-29 16:29:13 on dist-test-slave-dssx"
server_key: "3841b75a6bc07b546937e2d0a8d52ba8"
server_key_iv: "ac936998cf33d29c717a8df4cc5116e1"
server_key_version: "encryptionkey@0"
I20260929 16:29:13.088035 10142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:29:13.091742 10156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:13.093158 10142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:29:13.093365 10142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "063f664529a9454198671b61f262e224"
format_stamp: "Formatted at 2026-09-29 16:29:13 on dist-test-slave-dssx"
server_key: "3841b75a6bc07b546937e2d0a8d52ba8"
server_key_iv: "ac936998cf33d29c717a8df4cc5116e1"
server_key_version: "encryptionkey@0"
I20260929 16:29:13.093665 10142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:13.119371 10142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:13.123746 10142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:13.124032 10142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:13.142742 10142 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:35201
I20260929 16:29:13.142694 10207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:35201 every 8 connection(s)
I20260929 16:29:13.143708 10142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:13.147723 10208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:13.153110 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 10142
I20260929 16:29:13.153340 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:13.153986 6050 external_mini_cluster.cc:1428] Setting key 126b9d7041ea517e431dc8fa82ff0182
I20260929 16:29:13.159157 10208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224: Bootstrap starting.
I20260929 16:29:13.163105 10208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:13.164494 10208 log.cc:824] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:13.167670 10208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224: No bootstrap required, opened a new log
Sep 29 16:29:13 dist-test-slave-dssx krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:13.175802 10208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f664529a9454198671b61f262e224" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35201 } }
I20260929 16:29:13.176312 10208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:13.176447 10208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063f664529a9454198671b61f262e224, State: Initialized, Role: FOLLOWER
I20260929 16:29:13.177232 10208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f664529a9454198671b61f262e224" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35201 } }
I20260929 16:29:13.177620 10208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:13.177659 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:13.156147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.5.232.190:35201)
0929 16:29:13.157059 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:13.157071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:13.157115 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:29:13.158822 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:13.158847 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:13.159260 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:13.159674 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:13.161612 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:13.162949 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:13.164871 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:13.165695 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:13.165907 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:13.169842 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:13.169849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:13.169865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:13.169905 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:13.173034 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:13.173823 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:13.173830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:13.173837 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:13.174030 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:13.174445 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:13.174448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:13.174450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:13.175012 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:13.175314 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:13.175745 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 16:29:13.176222 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":396,"threads_started":1}
I20260929 16:29:13.177796 10208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:13.177981 10208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:13.180436 10208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f664529a9454198671b61f262e224" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35201 } }
I20260929 16:29:13.180977 10208 leader_election.cc:302] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 063f664529a9454198671b61f262e224; no voters:
I20260929 16:29:13.181595 10208 leader_election.cc:288] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:13.181735 10213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:13.182410 10213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [term 1 LEADER]: Becoming Leader. State: Replica: 063f664529a9454198671b61f262e224, State: Running, Role: LEADER
I20260929 16:29:13.182919 10213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f664529a9454198671b61f262e224" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35201 } }
I20260929 16:29:13.183705 10208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:13.186070 10214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "063f664529a9454198671b61f262e224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f664529a9454198671b61f262e224" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35201 } } }
I20260929 16:29:13.186429 10214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:13.186175 10215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 063f664529a9454198671b61f262e224. Latest consensus state: current_term: 1 leader_uuid: "063f664529a9454198671b61f262e224" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f664529a9454198671b61f262e224" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35201 } } }
I20260929 16:29:13.186621 10215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:13.188022 10218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:13.194844 10218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:13.204631 10218 catalog_manager.cc:1347] Generated new cluster ID: 42723f6d2fc04d79b9dd274976a22519
I20260929 16:29:13.204761 10218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:13.236471 10218 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:13.238207 10218 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:13.257120 10218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224: Generated new TSK 0
I20260929 16:29:13.258090 10218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:29:13.308754 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35201
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:13.643196 10234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:13.643508 10234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:13.643678 10234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:13.651142 10234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:13.651260 10234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:13.651443 10234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:13.661280 10234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:13.662681 10234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:13.664032 10234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:13.675977 10239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:13.676738 10240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:13.678164 10242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:13.678896 10234 server_base.cc:1034] running on GCE node
I20260929 16:29:13.679536 10234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:13.680501 10234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:13.681695 10234 hybrid_clock.cc:648] HybridClock initialized: now 1790699353681654 us; error 42 us; skew 500 ppm
I20260929 16:29:13.682029 10234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:13 dist-test-slave-dssx krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:13.685213 10234 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:29:13.686522 10234 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:13.686618 10234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:13.687004 10234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:13.690474 10234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "c7a93771d33249e88faf42abf122195a"
format_stamp: "Formatted at 2026-09-29 16:29:13 on dist-test-slave-dssx"
server_key: "387ff555ebdf6d4e5efdffd5353d39a2"
server_key_iv: "522c8026c757eb7cfbaaf477307ef61e"
server_key_version: "encryptionkey@0"
I20260929 16:29:13.691287 10234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "c7a93771d33249e88faf42abf122195a"
format_stamp: "Formatted at 2026-09-29 16:29:13 on dist-test-slave-dssx"
server_key: "387ff555ebdf6d4e5efdffd5353d39a2"
server_key_iv: "522c8026c757eb7cfbaaf477307ef61e"
server_key_version: "encryptionkey@0"
I20260929 16:29:13.697101 10234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:29:13.701067 10248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:13.702593 10234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:13.702793 10234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "c7a93771d33249e88faf42abf122195a"
format_stamp: "Formatted at 2026-09-29 16:29:13 on dist-test-slave-dssx"
server_key: "387ff555ebdf6d4e5efdffd5353d39a2"
server_key_iv: "522c8026c757eb7cfbaaf477307ef61e"
server_key_version: "encryptionkey@0"
I20260929 16:29:13.703047 10234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:13.720467 10234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:13.726828 10234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:13.727147 10234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:13.728148 10234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:13.729592 10234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:13.729707 10234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:13.729838 10234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:13.729939 10234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:13.760942 10234 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:42979
I20260929 16:29:13.760988 10360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:42979 every 8 connection(s)
I20260929 16:29:13.761883 10234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:13.768751 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 10234
I20260929 16:29:13.768972 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:13.769773 6050 external_mini_cluster.cc:1428] Setting key 1255df7fc1f5476474d7d5ff1f171388
Sep 29 16:29:13 dist-test-slave-dssx krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:13.783531 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:13.764258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:54255 (local address 127.5.232.190:35201)
0929 16:29:13.764688 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:13.764695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:13.765796 (+ 1101us) server_negotiation.cc:407] Connection header received
0929 16:29:13.767650 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:13.767654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:13.767725 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:13.767827 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:13.769618 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:13.770606 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:13.772342 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:13.772746 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:13.772882 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:13.776664 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:13.776674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:13.776679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:13.776723 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:13.780241 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:13.780972 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:13.780981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:13.780987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:13.781127 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:13.781574 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:13.781581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:13.781587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:13.781962 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:13.782110 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:13.782781 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 16:29:13.783026 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":208,"threads_started":1}
I20260929 16:29:13.784215 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:13.764874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35201 (local address 127.5.232.129:54255)
0929 16:29:13.765633 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:13.765673 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:13.767371 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:13.768052 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:13.768061 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:13.768626 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:13.769347 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:13.769367 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:13.770769 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:13.770774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:13.772190 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:13.772202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:13.772987 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:13.772995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:13.773206 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:13.773928 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:13.773963 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:13.776426 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:13.780389 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:13.780396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:13.780410 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:13.780838 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:13.781228 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:13.781231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:13.781233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:13.781460 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:13.782077 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:13.782082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:13.782396 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 16:29:13.782620 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 16:29:13.783023 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":277,"threads_started":1}
I20260929 16:29:13.785960 10361 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35201
I20260929 16:29:13.786263 10361 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:13.786903 10361 heartbeater.cc:511] Master 127.5.232.190:35201 requested a full tablet report, sending...
I20260929 16:29:13.789408 10173 ts_manager.cc:194] Registered new tserver with Master: c7a93771d33249e88faf42abf122195a (127.5.232.129:42979)
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
I20260929 16:29:13.791062 10173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:54255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:29:13.815655 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35201
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:14.143625 10367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:14.143893 10367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:14.144066 10367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:14.151396 10367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:14.151503 10367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:14.151691 10367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:14.161497 10367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:14.162946 10367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:14.164319 10367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:14.177345 10372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:14.177266 10375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:14.177311 10373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:14.178136 10367 server_base.cc:1034] running on GCE node
I20260929 16:29:14.178860 10367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:14.179908 10367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:14.181141 10367 hybrid_clock.cc:648] HybridClock initialized: now 1790699354181094 us; error 42 us; skew 500 ppm
I20260929 16:29:14.181532 10367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:14 dist-test-slave-dssx krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:14.184803 10367 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:29:14.186049 10367 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:14.186175 10367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:14.186501 10367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:14.189538 10367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "fbad9d4afd50499182a014aafb85ab12"
format_stamp: "Formatted at 2026-09-29 16:29:14 on dist-test-slave-dssx"
server_key: "bd5b6409ad207acdba2e3320562c1ef6"
server_key_iv: "8b6554b47b4bad1cb7951fb4b82f874c"
server_key_version: "encryptionkey@0"
I20260929 16:29:14.190361 10367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "fbad9d4afd50499182a014aafb85ab12"
format_stamp: "Formatted at 2026-09-29 16:29:14 on dist-test-slave-dssx"
server_key: "bd5b6409ad207acdba2e3320562c1ef6"
server_key_iv: "8b6554b47b4bad1cb7951fb4b82f874c"
server_key_version: "encryptionkey@0"
I20260929 16:29:14.195467 10367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:29:14.199043 10381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:14.200369 10367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:29:14.200551 10367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "fbad9d4afd50499182a014aafb85ab12"
format_stamp: "Formatted at 2026-09-29 16:29:14 on dist-test-slave-dssx"
server_key: "bd5b6409ad207acdba2e3320562c1ef6"
server_key_iv: "8b6554b47b4bad1cb7951fb4b82f874c"
server_key_version: "encryptionkey@0"
I20260929 16:29:14.200771 10367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:14.228809 10367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:14.232951 10367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:14.233251 10367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:14.234362 10367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:14.235705 10367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:14.235829 10367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:14.235963 10367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:14.236045 10367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:14.269915 10367 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:33817
I20260929 16:29:14.270002 10493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:33817 every 8 connection(s)
I20260929 16:29:14.270761 10367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:14.274652 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 10367
I20260929 16:29:14.274813 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:14.275367 6050 external_mini_cluster.cc:1428] Setting key 97714e23870a50e79004190a7c0634dc
Sep 29 16:29:14 dist-test-slave-dssx krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:14.294431 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.273211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:48099 (local address 127.5.232.190:35201)
0929 16:29:14.273408 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:14.273416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:14.274669 (+ 1253us) server_negotiation.cc:407] Connection header received
0929 16:29:14.276907 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:14.276912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:14.277003 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:14.277127 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:14.279115 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.280028 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.282508 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.283156 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.283287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.287499 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:14.287508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:14.287516 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:14.287559 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:14.291126 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:14.291854 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:14.291862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:14.291867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:14.292032 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:14.292448 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:14.292453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:14.292457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:14.292859 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:14.293025 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:14.293756 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 16:29:14.294009 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:29:14.295132 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.273718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35201 (local address 127.5.232.130:48099)
0929 16:29:14.274473 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:14.274526 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:14.276650 (+ 2124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:14.277380 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:14.277409 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:14.278123 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:14.278925 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.278952 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.280194 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.280201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.282373 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.282389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.283608 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.283617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.283935 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.284658 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:14.284691 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:14.287247 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:14.291264 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:14.291271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:14.291286 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:14.291710 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:14.292148 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:14.292151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:14.292154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:14.292321 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:14.292991 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:14.292997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:14.293322 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 16:29:14.293606 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 16:29:14.294014 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":273,"threads_started":1}
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
I20260929 16:29:14.296733 10494 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35201
I20260929 16:29:14.297073 10494 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:14.297771 10494 heartbeater.cc:511] Master 127.5.232.190:35201 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
I20260929 16:29:14.299403 10173 ts_manager.cc:194] Registered new tserver with Master: fbad9d4afd50499182a014aafb85ab12 (127.5.232.130:33817)
I20260929 16:29:14.300685 10173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:48099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:29:14.321923 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35201
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:14.646111 10499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:14.646394 10499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:14.646560 10499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:14.653962 10499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:14.654081 10499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:14.654270 10499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:14.664007 10499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:14.665349 10499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:14.666747 10499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:14.678395 10504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:14.679608 10505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:14.679982 10507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:14.680447 10499 server_base.cc:1034] running on GCE node
I20260929 16:29:14.681003 10499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:14.682016 10499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:14.683259 10499 hybrid_clock.cc:648] HybridClock initialized: now 1790699354683216 us; error 50 us; skew 500 ppm
I20260929 16:29:14.683594 10499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:14 dist-test-slave-dssx krb5kdc[10129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:14.686902 10499 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:29:14.688148 10499 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:14.688283 10499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:14.688607 10499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:14.691798 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "7f7898f36cf74122a425c0a396f243a1"
format_stamp: "Formatted at 2026-09-29 16:29:14 on dist-test-slave-dssx"
server_key: "d02a5e5964ca4eb475c3c6728d2ef1a8"
server_key_iv: "0a3905813ff6a0eb12e776ad39043cc5"
server_key_version: "encryptionkey@0"
I20260929 16:29:14.692620 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "7f7898f36cf74122a425c0a396f243a1"
format_stamp: "Formatted at 2026-09-29 16:29:14 on dist-test-slave-dssx"
server_key: "d02a5e5964ca4eb475c3c6728d2ef1a8"
server_key_iv: "0a3905813ff6a0eb12e776ad39043cc5"
server_key_version: "encryptionkey@0"
I20260929 16:29:14.697819 10499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:29:14.701200 10513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:14.702523 10499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:14.702710 10499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "7f7898f36cf74122a425c0a396f243a1"
format_stamp: "Formatted at 2026-09-29 16:29:14 on dist-test-slave-dssx"
server_key: "d02a5e5964ca4eb475c3c6728d2ef1a8"
server_key_iv: "0a3905813ff6a0eb12e776ad39043cc5"
server_key_version: "encryptionkey@0"
I20260929 16:29:14.702939 10499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:14.716938 10499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:14.721633 10499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:14.721931 10499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:14.722918 10499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:14.724287 10499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:14.724419 10499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:14.724555 10499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:14.724642 10499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:14.757319 10499 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:45321
I20260929 16:29:14.757367 10625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:45321 every 8 connection(s)
I20260929 16:29:14.758239 10499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:14.761590 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 10499
I20260929 16:29:14.761785 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:14.762326 6050 external_mini_cluster.cc:1428] Setting key fa0074734ee0649e5fe9ec58a704db82
Sep 29 16:29:14 dist-test-slave-dssx krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:14.778143 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.760683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:58957 (local address 127.5.232.190:35201)
0929 16:29:14.760852 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:14.760856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:14.762376 (+ 1520us) server_negotiation.cc:407] Connection header received
0929 16:29:14.764731 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:14.764737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:14.764815 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:14.764927 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:14.766604 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.767364 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.768842 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.769244 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.769339 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.772655 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:14.772660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:14.772664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:14.772696 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:14.775111 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:14.775800 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:14.775804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:14.775806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:14.775926 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:14.776376 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:14.776381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:14.776384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:14.776677 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:14.776838 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:14.777585 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 16:29:14.777795 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:29:14.779121 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.761268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35201 (local address 127.5.232.131:58957)
0929 16:29:14.762188 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:14.762241 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:14.764429 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:14.765190 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:14.765201 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:14.765866 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:14.766424 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.766441 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.767488 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.767492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.768708 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.768719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.769490 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.769496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.769676 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.770298 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:14.770323 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:14.772457 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:14.775239 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:14.775245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:14.775258 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:14.775671 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:14.776033 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:14.776038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:14.776041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:14.776253 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:14.776812 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:14.776821 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:14.777172 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 16:29:14.777473 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 16:29:14.778008 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":350,"threads_started":1}
I20260929 16:29:14.780696 10626 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35201
I20260929 16:29:14.781018 10626 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:14.781730 10626 heartbeater.cc:511] Master 127.5.232.190:35201 requested a full tablet report, sending...
I20260929 16:29:14.783149 10173 ts_manager.cc:194] Registered new tserver with Master: 7f7898f36cf74122a425c0a396f243a1 (127.5.232.131:45321)
I20260929 16:29:14.784029 10173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:58957
I20260929 16:29:14.790248 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:29:14.794898 10361 heartbeater.cc:503] Master 127.5.232.190:35201 was elected leader, sending a full tablet report...
I20260929 16:29:14.808113 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.794888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.5.232.190:35201)
0929 16:29:14.795106 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:14.795115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:14.795737 (+ 622us) server_negotiation.cc:407] Connection header received
0929 16:29:14.795924 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:14.795929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:14.795995 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:14.796105 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:14.797557 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.798330 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.799504 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.799933 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.800027 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.802338 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:14.802343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:14.802346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:14.802381 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:14.805118 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:14.805959 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:14.805963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:14.805965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:14.806095 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:14.806647 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:14.806651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:14.806654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:14.807078 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:14.807266 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:14.807601 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:29:14.807825 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:29:14.812516 10173 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:14.819727 10173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:14.849789 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.832007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:33817 (local address 127.0.0.1:43100)
0929 16:29:14.834371 (+ 2364us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:14.834413 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:14.834646 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:14.837087 (+ 2441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:14.837098 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:14.837140 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:14.837632 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.837650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.841627 (+ 3977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.841635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.846439 (+ 4804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.846464 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.848721 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.848727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.848953 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.849013 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 16:29:14.849137 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:29:14.849330 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"thread_start_us":2016,"threads_started":1}
I20260929 16:29:14.850528 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.832873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.5.232.130:33817)
0929 16:29:14.833827 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:14.833840 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:14.834506 (+ 666us) server_negotiation.cc:407] Connection header received
0929 16:29:14.834925 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:14.834933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:14.835216 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:14.836643 (+ 1427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:14.837875 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.841355 (+ 3480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.846703 (+ 5348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.848206 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.848355 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.848612 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:14.850015 (+ 1403us) server_negotiation.cc:299] Negotiation successful
0929 16:29:14.850208 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":804,"threads_started":1}
I20260929 16:29:14.854547 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.832270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:45321 (local address 127.0.0.1:34902)
0929 16:29:14.834989 (+ 2719us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:14.835005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:14.835097 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:14.846793 (+ 11696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:14.846798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:14.846827 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:14.848093 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.848112 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.850175 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.850180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.852228 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.852240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.853605 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.853612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.853826 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.853859 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:29:14.853986 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 16:29:14.854137 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2608,"mutex_wait_us":26,"thread_start_us":2492,"threads_started":1}
I20260929 16:29:14.854539 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.833339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.5.232.131:45321)
0929 16:29:14.845950 (+ 12611us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:14.845964 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:14.846012 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:29:14.846145 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:14.846158 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:14.846545 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:14.846825 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:14.848345 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.850011 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.852402 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.853331 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.853551 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.853801 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:14.854036 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 16:29:14.854195 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12356,"thread_start_us":13698,"threads_started":1}
I20260929 16:29:14.854902 10429 tablet_service.cc:1467] Processing CreateTablet for tablet ad7cd558344d4d89ad2b6d82983f8221 (DEFAULT_TABLE table=test-table [id=229dfba594a34d189fc8667449aa8800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:14.856626 10429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7cd558344d4d89ad2b6d82983f8221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:14.857203 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.835440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:42979 (local address 127.0.0.1:44934)
0929 16:29:14.848329 (+ 12889us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:14.848350 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:14.848498 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:14.849450 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:14.849454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:14.849477 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:14.850136 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.850153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.852141 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.852146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.854176 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.854190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.856382 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.856388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.856569 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.856603 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:29:14.856703 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:29:14.856846 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12744,"mutex_wait_us":3,"thread_start_us":12572,"threads_started":1}
I20260929 16:29:14.857496 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.835591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.5.232.129:42979)
0929 16:29:14.847660 (+ 12069us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:14.847666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:14.848456 (+ 790us) server_negotiation.cc:407] Connection header received
0929 16:29:14.848666 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:14.848676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:14.848952 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:14.849165 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:14.850400 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.851977 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.854378 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.856138 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.856296 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.856542 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:14.856797 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 16:29:14.856974 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11926,"thread_start_us":11775,"threads_started":1}
I20260929 16:29:14.858886 10561 tablet_service.cc:1467] Processing CreateTablet for tablet ad7cd558344d4d89ad2b6d82983f8221 (DEFAULT_TABLE table=test-table [id=229dfba594a34d189fc8667449aa8800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:14.860353 10561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7cd558344d4d89ad2b6d82983f8221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:14.861183 10296 tablet_service.cc:1467] Processing CreateTablet for tablet ad7cd558344d4d89ad2b6d82983f8221 (DEFAULT_TABLE table=test-table [id=229dfba594a34d189fc8667449aa8800]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:14.862663 10296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7cd558344d4d89ad2b6d82983f8221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:14.868930 10644 tablet_bootstrap.cc:492] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: Bootstrap starting.
I20260929 16:29:14.872633 10644 tablet_bootstrap.cc:654] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:14.874301 10644 log.cc:824] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:14.875479 10645 tablet_bootstrap.cc:492] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: Bootstrap starting.
I20260929 16:29:14.878376 10647 tablet_bootstrap.cc:492] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: Bootstrap starting.
I20260929 16:29:14.879894 10645 tablet_bootstrap.cc:654] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:14.880224 10644 tablet_bootstrap.cc:492] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: No bootstrap required, opened a new log
I20260929 16:29:14.880538 10644 ts_tablet_manager.cc:1397] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260929 16:29:14.881278 10645 log.cc:824] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:14.882310 10647 tablet_bootstrap.cc:654] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:14.883852 10647 log.cc:824] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:14.887099 10647 tablet_bootstrap.cc:492] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: No bootstrap required, opened a new log
I20260929 16:29:14.887419 10647 ts_tablet_manager.cc:1397] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 16:29:14.887539 10645 tablet_bootstrap.cc:492] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: No bootstrap required, opened a new log
I20260929 16:29:14.887820 10645 ts_tablet_manager.cc:1397] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260929 16:29:14.891321 10644 raft_consensus.cc:348] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } }
I20260929 16:29:14.891829 10644 raft_consensus.cc:374] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:14.891942 10644 raft_consensus.cc:729] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbad9d4afd50499182a014aafb85ab12, State: Initialized, Role: FOLLOWER
I20260929 16:29:14.892630 10644 consensus_queue.cc:260] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } }
I20260929 16:29:14.893867 10494 heartbeater.cc:503] Master 127.5.232.190:35201 was elected leader, sending a full tablet report...
I20260929 16:29:14.894631 10644 ts_tablet_manager.cc:1428] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 16:29:14.896482 10647 raft_consensus.cc:348] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } }
I20260929 16:29:14.896952 10647 raft_consensus.cc:374] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:14.897055 10647 raft_consensus.cc:729] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a93771d33249e88faf42abf122195a, State: Initialized, Role: FOLLOWER
I20260929 16:29:14.897831 10647 consensus_queue.cc:260] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } }
I20260929 16:29:14.898634 10645 raft_consensus.cc:348] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } }
I20260929 16:29:14.899065 10645 raft_consensus.cc:374] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:14.899156 10645 raft_consensus.cc:729] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7898f36cf74122a425c0a396f243a1, State: Initialized, Role: FOLLOWER
I20260929 16:29:14.899773 10645 consensus_queue.cc:260] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } }
I20260929 16:29:14.901252 10647 ts_tablet_manager.cc:1428] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 16:29:14.902170 10626 heartbeater.cc:503] Master 127.5.232.190:35201 was elected leader, sending a full tablet report...
I20260929 16:29:14.902891 10645 ts_tablet_manager.cc:1428] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 16:29:14.944109 10652 raft_consensus.cc:482] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:14.944468 10652 raft_consensus.cc:504] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } }
I20260929 16:29:14.946476 10652 leader_election.cc:288] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbad9d4afd50499182a014aafb85ab12 (127.5.232.130:33817), c7a93771d33249e88faf42abf122195a (127.5.232.129:42979)
I20260929 16:29:14.953042 10628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.946932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:33817 (local address 127.5.232.131:43249)
0929 16:29:14.947287 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:14.947311 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:14.947494 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:14.947940 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:14.947947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:14.947977 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:14.948175 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.948187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.949425 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.949431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.950910 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.950922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.952247 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.952253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.952452 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.952489 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:29:14.952599 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:29:14.952735 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260929 16:29:14.953032 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.947369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:43249 (local address 127.5.232.130:33817)
0929 16:29:14.947582 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:14.947590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:14.947611 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:29:14.947693 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:14.947701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:14.947775 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:14.947889 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:14.948326 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.949273 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.951083 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.952100 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.952218 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.952270 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:14.952631 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 16:29:14.952724 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:29:14.953755 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.947035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:50869 (local address 127.5.232.129:42979)
0929 16:29:14.947296 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:14.947303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:14.947919 (+ 616us) server_negotiation.cc:407] Connection header received
0929 16:29:14.948065 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:14.948071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:14.948165 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:14.948266 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:14.948920 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.949846 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.951749 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.952714 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.952867 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.952942 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:14.953312 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 16:29:14.953521 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 16:29:14.954315 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.947142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:42979 (local address 127.5.232.131:50869)
0929 16:29:14.947806 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:14.947826 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:14.947943 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:14.948310 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:14.948314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:14.948337 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:14.948741 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.948752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.949999 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.950003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.951589 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:14.951603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.952916 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.952922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:14.953112 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.953146 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:29:14.953941 (+ 795us) client_negotiation.cc:240] Negotiation successful
0929 16:29:14.954066 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":3,"thread_start_us":386,"threads_started":1}
I20260929 16:29:14.954190 10449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7cd558344d4d89ad2b6d82983f8221" candidate_uuid: "7f7898f36cf74122a425c0a396f243a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbad9d4afd50499182a014aafb85ab12" is_pre_election: true
I20260929 16:29:14.954716 10449 raft_consensus.cc:2454] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f7898f36cf74122a425c0a396f243a1 in term 0.
I20260929 16:29:14.955235 10316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7cd558344d4d89ad2b6d82983f8221" candidate_uuid: "7f7898f36cf74122a425c0a396f243a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a93771d33249e88faf42abf122195a" is_pre_election: true
I20260929 16:29:14.955467 10517 leader_election.cc:302] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f7898f36cf74122a425c0a396f243a1, fbad9d4afd50499182a014aafb85ab12; no voters:
I20260929 16:29:14.955750 10316 raft_consensus.cc:2454] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f7898f36cf74122a425c0a396f243a1 in term 0.
I20260929 16:29:14.955900 10652 raft_consensus.cc:2781] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:14.956041 10652 raft_consensus.cc:482] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:14.956147 10652 raft_consensus.cc:3037] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:14.958439 10652 raft_consensus.cc:504] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } }
I20260929 16:29:14.959036 10652 leader_election.cc:288] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [CANDIDATE]: Term 1 election: Requested vote from peers fbad9d4afd50499182a014aafb85ab12 (127.5.232.130:33817), c7a93771d33249e88faf42abf122195a (127.5.232.129:42979)
I20260929 16:29:14.959508 10316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7cd558344d4d89ad2b6d82983f8221" candidate_uuid: "7f7898f36cf74122a425c0a396f243a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7a93771d33249e88faf42abf122195a"
I20260929 16:29:14.959514 10449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7cd558344d4d89ad2b6d82983f8221" candidate_uuid: "7f7898f36cf74122a425c0a396f243a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbad9d4afd50499182a014aafb85ab12"
I20260929 16:29:14.959715 10316 raft_consensus.cc:3037] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:14.959722 10449 raft_consensus.cc:3037] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:14.961934 10316 raft_consensus.cc:2454] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f7898f36cf74122a425c0a396f243a1 in term 1.
I20260929 16:29:14.961963 10449 raft_consensus.cc:2454] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f7898f36cf74122a425c0a396f243a1 in term 1.
I20260929 16:29:14.962405 10514 leader_election.cc:302] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f7898f36cf74122a425c0a396f243a1, c7a93771d33249e88faf42abf122195a; no voters:
I20260929 16:29:14.962708 10652 raft_consensus.cc:2781] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:14.963177 10652 raft_consensus.cc:686] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7898f36cf74122a425c0a396f243a1, State: Running, Role: LEADER
I20260929 16:29:14.963665 10652 consensus_queue.cc:237] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } }
I20260929 16:29:14.968032 10173 catalog_manager.cc:5495] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f7898f36cf74122a425c0a396f243a1 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "7f7898f36cf74122a425c0a396f243a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f7898f36cf74122a425c0a396f243a1" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 } health_report { overall_health: UNKNOWN } } }
I20260929 16:29:14.997907 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:14.992817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.5.232.131:45321)
0929 16:29:14.993027 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:14.993035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:14.993183 (+ 148us) server_negotiation.cc:407] Connection header received
0929 16:29:14.993358 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:14.993362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:14.993494 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:14.993626 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:14.993936 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.994631 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.995881 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:14.996246 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:14.996385 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:14.996758 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:14.997275 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:14.997434 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:14.997548 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 16:29:14.997612 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:29:15.005925 10316 raft_consensus.cc:1261] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [term 1 FOLLOWER]: Refusing update from remote peer 7f7898f36cf74122a425c0a396f243a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:15.005926 10449 raft_consensus.cc:1261] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [term 1 FOLLOWER]: Refusing update from remote peer 7f7898f36cf74122a425c0a396f243a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:15.006969 10654 consensus_queue.cc:1035] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7a93771d33249e88faf42abf122195a" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:15.007398 10652 consensus_queue.cc:1035] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbad9d4afd50499182a014aafb85ab12" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:29:15.011566 10627 tablet.cc:2369] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:29:15.014113 10362 tablet.cc:2369] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:29:15.022573 10495 tablet.cc:2369] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:29:15.032380 10173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:15.032670 10173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:15.036919 10173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 063f664529a9454198671b61f262e224: Sending DeleteTablet for 3 replicas of tablet ad7cd558344d4d89ad2b6d82983f8221
I20260929 16:29:15.038216 10561 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7cd558344d4d89ad2b6d82983f8221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:15 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:34902
I20260929 16:29:15.038471 10296 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7cd558344d4d89ad2b6d82983f8221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:15 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:44934
I20260929 16:29:15.038498 10295 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7cd558344d4d89ad2b6d82983f8221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:15 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:44934
I20260929 16:29:15.038910 10429 tablet_service.cc:1514] Processing DeleteTablet for tablet ad7cd558344d4d89ad2b6d82983f8221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:15 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:43100
I20260929 16:29:15.042383 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 10234
I20260929 16:29:15.051008 10668 tablet_replica.cc:330] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: stopping tablet replica
I20260929 16:29:15.051532 10668 raft_consensus.cc:2229] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:15.052105 10668 raft_consensus.cc:2258] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:15.052994 10669 tablet_replica.cc:330] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: stopping tablet replica
I20260929 16:29:15.053745 10669 raft_consensus.cc:2229] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:15.054283 10669 raft_consensus.cc:2258] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:15.055313 10668 ts_tablet_manager.cc:1905] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:15.056406 10669 ts_tablet_manager.cc:1905] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:15.062655 10668 ts_tablet_manager.cc:1918] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:15.062944 10668 log.cc:1196] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/ad7cd558344d4d89ad2b6d82983f8221
I20260929 16:29:15.063583 10668 ts_tablet_manager.cc:1939] T ad7cd558344d4d89ad2b6d82983f8221 P c7a93771d33249e88faf42abf122195a: Deleting consensus metadata
I20260929 16:29:15.064934 10669 ts_tablet_manager.cc:1918] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:15.065105 10669 log.cc:1196] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/ad7cd558344d4d89ad2b6d82983f8221
I20260929 16:29:15.065804 10157 catalog_manager.cc:4841] TS c7a93771d33249e88faf42abf122195a (127.5.232.129:42979): tablet ad7cd558344d4d89ad2b6d82983f8221 (table test-table [id=229dfba594a34d189fc8667449aa8800]) successfully deleted
W20260929 16:29:15.066445 10157 catalog_manager.cc:4808] TS c7a93771d33249e88faf42abf122195a (127.5.232.129:42979): delete failed for tablet ad7cd558344d4d89ad2b6d82983f8221 because the tablet was not found. No further retry: Not found: Tablet not found: ad7cd558344d4d89ad2b6d82983f8221
I20260929 16:29:15.066965 10669 ts_tablet_manager.cc:1939] T ad7cd558344d4d89ad2b6d82983f8221 P fbad9d4afd50499182a014aafb85ab12: Deleting consensus metadata
I20260929 16:29:15.068404 10667 tablet_replica.cc:330] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: stopping tablet replica
I20260929 16:29:15.068795 10160 catalog_manager.cc:4841] TS fbad9d4afd50499182a014aafb85ab12 (127.5.232.130:33817): tablet ad7cd558344d4d89ad2b6d82983f8221 (table test-table [id=229dfba594a34d189fc8667449aa8800]) successfully deleted
I20260929 16:29:15.068876 10667 raft_consensus.cc:2229] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:29:15.069535 10667 raft_consensus.cc:2258] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:15.071498 10667 ts_tablet_manager.cc:1905] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:29:15 dist-test-slave-dssx krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:29:15.077920 10667 ts_tablet_manager.cc:1918] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:15.078114 10667 log.cc:1196] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/ad7cd558344d4d89ad2b6d82983f8221
I20260929 16:29:15.078716 10667 ts_tablet_manager.cc:1939] T ad7cd558344d4d89ad2b6d82983f8221 P 7f7898f36cf74122a425c0a396f243a1: Deleting consensus metadata
I20260929 16:29:15.080332 10157 catalog_manager.cc:4841] TS 7f7898f36cf74122a425c0a396f243a1 (127.5.232.131:45321): tablet ad7cd558344d4d89ad2b6d82983f8221 (table test-table [id=229dfba594a34d189fc8667449aa8800]) successfully deleted
I20260929 16:29:15.084916 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:15.044014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.5.232.129:42979)
0929 16:29:15.044243 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:15.044253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:15.065332 (+ 21079us) server_negotiation.cc:407] Connection header received
0929 16:29:15.065881 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:15.065887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:15.065969 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:15.066063 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:15.067914 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:15.069054 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:15.070905 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:15.071595 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:15.071731 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:15.077652 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:15.077672 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:15.077693 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:15.077743 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:15.081703 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:15.082451 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:15.082461 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:15.082465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:15.082636 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:15.083005 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:15.083010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:15.083013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:15.083507 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:15.083841 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:15.084282 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 16:29:15.084531 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:29:15.085698 10356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:15.216912 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 10234
I20260929 16:29:15.234488 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 10367
Sep 29 16:29:15 dist-test-slave-dssx krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:29:15.252640 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:15.235339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.5.232.130:33817)
0929 16:29:15.235525 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:15.235540 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:15.235562 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:29:15.235722 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:15.235730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:15.235822 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:15.235941 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:15.237592 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:15.238738 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:15.240038 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:15.240671 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:15.240799 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:15.246177 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:15.246196 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:15.246219 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:15.246268 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:15.249934 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:15.250612 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:15.250622 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:15.250625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:15.250777 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:15.251104 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:15.251109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:15.251112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:15.251567 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:15.251851 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:15.252042 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 16:29:15.252267 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:29:15.253324 10489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:15.343752 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 10367
I20260929 16:29:15.361040 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 10499
Sep 29 16:29:15 dist-test-slave-dssx krb5kdc[10129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:29:15.379415 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:15.362061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.5.232.131:45321)
0929 16:29:15.362227 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:15.362231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:15.362248 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:29:15.362346 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:15.362350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:15.362410 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:15.362506 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:15.364235 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:15.365031 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:15.367105 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:15.367574 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:15.367678 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:15.373246 (+ 5568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:15.373260 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:15.373277 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:15.373316 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:15.376429 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:15.377275 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:15.377282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:15.377284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:15.377537 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:15.377950 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:15.377954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:15.377956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:15.378303 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:15.378536 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:15.378895 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 16:29:15.379056 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:29:15.380187 10621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:15.501886 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 10499
I20260929 16:29:15.521011 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 10142
I20260929 16:29:15.522181 10203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:15.629141 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 10142
2026-09-29T16:29:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4222 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T16:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:15Z Disabled control of system clock
I20260929 16:29:15.698278 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42781
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:29:16.036837 10678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:16.037154 10678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:16.037240 10678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:16.044472 10678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:16.044591 10678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:16.044631 10678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:16.044678 10678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:16.044723 10678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:16.053894 10678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:16.055114 10678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:16.056897 10678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:16.068498 10684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:16.068493 10686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:16.068533 10683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:16.069247 10678 server_base.cc:1034] running on GCE node
I20260929 16:29:16.070138 10678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:16.071660 10678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:16.072865 10678 hybrid_clock.cc:648] HybridClock initialized: now 1790699356072832 us; error 29 us; skew 500 ppm
I20260929 16:29:16.073227 10678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:29:16.079276 10678 webserver.cc:460] Webserver started at http://127.5.232.190:34429/ using document root <none> and password file <none>
I20260929 16:29:16.080039 10678 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:16.080158 10678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:16.080504 10678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:16.083645 10678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "253560e43dab4bd489ba824ca56f0329"
format_stamp: "Formatted at 2026-09-29 16:29:16 on dist-test-slave-dssx"
server_key: "ed43c8692db67b77b504bac7f69b7744"
server_key_iv: "2c40b15e9250a57f51dada8d50bba98d"
server_key_version: "encryptionkey@0"
I20260929 16:29:16.084470 10678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "253560e43dab4bd489ba824ca56f0329"
format_stamp: "Formatted at 2026-09-29 16:29:16 on dist-test-slave-dssx"
server_key: "ed43c8692db67b77b504bac7f69b7744"
server_key_iv: "2c40b15e9250a57f51dada8d50bba98d"
server_key_version: "encryptionkey@0"
I20260929 16:29:16.090176 10678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:29:16.094129 10692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:16.095520 10678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:16.095664 10678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "253560e43dab4bd489ba824ca56f0329"
format_stamp: "Formatted at 2026-09-29 16:29:16 on dist-test-slave-dssx"
server_key: "ed43c8692db67b77b504bac7f69b7744"
server_key_iv: "2c40b15e9250a57f51dada8d50bba98d"
server_key_version: "encryptionkey@0"
I20260929 16:29:16.095849 10678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:16.101339 10678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:16.102108 10678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:16.102346 10678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:16.119618 10678 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:42781
I20260929 16:29:16.119650 10743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:42781 every 8 connection(s)
I20260929 16:29:16.120986 10678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:16.124792 10744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:16.128279 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 10678
I20260929 16:29:16.128484 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:16.129106 6050 external_mini_cluster.cc:1428] Setting key c769e243079c515d9f2e90eddcb15d6e
I20260929 16:29:16.135437 10744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329: Bootstrap starting.
I20260929 16:29:16.137478 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:16.131161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.5.232.190:42781)
0929 16:29:16.132112 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:16.132126 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:16.132169 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:29:16.134069 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:16.134091 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:16.134474 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:16.134858 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:16.135902 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:16.135908 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:16.135921 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:16.135933 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:16.135987 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:16.136002 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:16.136217 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:16.136350 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 16:29:16.136595 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":349,"threads_started":1}
I20260929 16:29:16.138880 10744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:16.139918 10744 log.cc:824] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:16.142802 10744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329: No bootstrap required, opened a new log
I20260929 16:29:16.150190 10744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253560e43dab4bd489ba824ca56f0329" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42781 } }
I20260929 16:29:16.150503 10744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:16.150610 10744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253560e43dab4bd489ba824ca56f0329, State: Initialized, Role: FOLLOWER
I20260929 16:29:16.151172 10744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253560e43dab4bd489ba824ca56f0329" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42781 } }
I20260929 16:29:16.151367 10744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:16.151461 10744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:16.151588 10744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:16.154003 10744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253560e43dab4bd489ba824ca56f0329" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42781 } }
I20260929 16:29:16.154424 10744 leader_election.cc:302] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 253560e43dab4bd489ba824ca56f0329; no voters:
I20260929 16:29:16.154914 10744 leader_election.cc:288] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:16.155073 10749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:16.155740 10749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [term 1 LEADER]: Becoming Leader. State: Replica: 253560e43dab4bd489ba824ca56f0329, State: Running, Role: LEADER
I20260929 16:29:16.156273 10749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253560e43dab4bd489ba824ca56f0329" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42781 } }
I20260929 16:29:16.156987 10744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:16.159026 10750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "253560e43dab4bd489ba824ca56f0329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253560e43dab4bd489ba824ca56f0329" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42781 } } }
I20260929 16:29:16.159457 10750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:16.161578 10751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 253560e43dab4bd489ba824ca56f0329. Latest consensus state: current_term: 1 leader_uuid: "253560e43dab4bd489ba824ca56f0329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "253560e43dab4bd489ba824ca56f0329" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42781 } } }
I20260929 16:29:16.161844 10751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:16.162335 10755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:16.167428 10755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:16.175921 10755 catalog_manager.cc:1347] Generated new cluster ID: 9d92da3af3d34478b06547663a6d22a0
I20260929 16:29:16.176046 10755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:16.189205 10755 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:16.189677 10755 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:16.199308 10755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329: Generated new TSK 0
I20260929 16:29:16.200449 10755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:29:16.211696 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42781
--builtin_ntp_servers=127.5.232.148:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:29:16.547734 10768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:16.548079 10768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:16.548241 10768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:16.555554 10768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:16.555696 10768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:16.555868 10768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:16.565320 10768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:16.566768 10768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:16.568583 10768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:16.581656 10773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:16.581629 10774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:16.581703 10776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:16.582448 10768 server_base.cc:1034] running on GCE node
I20260929 16:29:16.583025 10768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:16.584082 10768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:16.585309 10768 hybrid_clock.cc:648] HybridClock initialized: now 1790699356585267 us; error 52 us; skew 500 ppm
I20260929 16:29:16.585651 10768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:29:16.587957 10768 webserver.cc:460] Webserver started at http://127.5.232.129:44125/ using document root <none> and password file <none>
I20260929 16:29:16.588660 10768 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:16.588789 10768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:16.589104 10768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:16.591967 10768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "33a164b75037433ab4f2e94652d0d553"
format_stamp: "Formatted at 2026-09-29 16:29:16 on dist-test-slave-dssx"
server_key: "38a43fbfbdb314899bf89bb89ea925d0"
server_key_iv: "6230611989b0849d12ff6c89bc90fba7"
server_key_version: "encryptionkey@0"
I20260929 16:29:16.592758 10768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "33a164b75037433ab4f2e94652d0d553"
format_stamp: "Formatted at 2026-09-29 16:29:16 on dist-test-slave-dssx"
server_key: "38a43fbfbdb314899bf89bb89ea925d0"
server_key_iv: "6230611989b0849d12ff6c89bc90fba7"
server_key_version: "encryptionkey@0"
I20260929 16:29:16.598007 10768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260929 16:29:16.601728 10782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:16.603052 10768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:16.603229 10768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "33a164b75037433ab4f2e94652d0d553"
format_stamp: "Formatted at 2026-09-29 16:29:16 on dist-test-slave-dssx"
server_key: "38a43fbfbdb314899bf89bb89ea925d0"
server_key_iv: "6230611989b0849d12ff6c89bc90fba7"
server_key_version: "encryptionkey@0"
I20260929 16:29:16.603492 10768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:16.609127 10768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:16.609953 10768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:16.610217 10768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:16.611083 10768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:16.612406 10768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:16.612537 10768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:16.612672 10768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:16.612761 10768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:16.645187 10768 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:36667
I20260929 16:29:16.645239 10894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:36667 every 8 connection(s)
I20260929 16:29:16.646561 10768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:16.652691 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 10768
I20260929 16:29:16.652858 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:16.653481 6050 external_mini_cluster.cc:1428] Setting key 128e159597993ea3b1d2b192b4830ffa
I20260929 16:29:16.655794 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:16.649149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:59807 (local address 127.5.232.190:42781)
0929 16:29:16.649610 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:16.649616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:16.650822 (+ 1206us) server_negotiation.cc:407] Connection header received
0929 16:29:16.653507 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:16.653511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:16.653601 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:16.653692 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:16.655095 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:16.655107 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:16.655111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:16.655127 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:16.655164 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:16.655195 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:16.655313 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:16.655541 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 16:29:16.655589 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":242,"threads_started":1}
I20260929 16:29:16.656479 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42781
--builtin_ntp_servers=127.5.232.148:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:29:16.657114 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:16.649765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42781 (local address 127.5.232.129:59807)
0929 16:29:16.650672 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:16.650708 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:16.652155 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:16.653974 (+ 1819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:16.653985 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:16.654110 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:16.654801 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:29:16.654804 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:16.654847 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:29:16.654930 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:16.655312 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:16.655321 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:16.655327 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:29:16.655619 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 16:29:16.655879 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":253,"threads_started":1}
I20260929 16:29:16.658700 10895 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42781
I20260929 16:29:16.659005 10895 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:16.659595 10895 heartbeater.cc:511] Master 127.5.232.190:42781 requested a full tablet report, sending...
I20260929 16:29:16.661725 10709 ts_manager.cc:194] Registered new tserver with Master: 33a164b75037433ab4f2e94652d0d553 (127.5.232.129:36667)
W20260929 16:29:16.974853 10899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:16.975128 10899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:16.975220 10899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:16.982438 10899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:16.982554 10899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:16.982666 10899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:16.991752 10899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:16.992997 10899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:16.994724 10899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:17.007732 10905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:17.007875 10904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:17.007921 10907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:17.009027 10899 server_base.cc:1034] running on GCE node
I20260929 16:29:17.009701 10899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:17.010592 10899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:17.011801 10899 hybrid_clock.cc:648] HybridClock initialized: now 1790699357011762 us; error 34 us; skew 500 ppm
I20260929 16:29:17.012151 10899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:29:17.014462 10899 webserver.cc:460] Webserver started at http://127.5.232.130:36975/ using document root <none> and password file <none>
I20260929 16:29:17.015156 10899 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:17.015297 10899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:17.015582 10899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:17.018419 10899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "846bee9149ca49a89738f57b329bd203"
format_stamp: "Formatted at 2026-09-29 16:29:17 on dist-test-slave-dssx"
server_key: "9e784cfd5dbd6d5ce23b8253c9ef4ea5"
server_key_iv: "b3af8b2e928bd6dbf98edb1b348edbe7"
server_key_version: "encryptionkey@0"
I20260929 16:29:17.019135 10899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "846bee9149ca49a89738f57b329bd203"
format_stamp: "Formatted at 2026-09-29 16:29:17 on dist-test-slave-dssx"
server_key: "9e784cfd5dbd6d5ce23b8253c9ef4ea5"
server_key_iv: "b3af8b2e928bd6dbf98edb1b348edbe7"
server_key_version: "encryptionkey@0"
I20260929 16:29:17.024442 10899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:29:17.027905 10913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:17.029198 10899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:17.029410 10899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "846bee9149ca49a89738f57b329bd203"
format_stamp: "Formatted at 2026-09-29 16:29:17 on dist-test-slave-dssx"
server_key: "9e784cfd5dbd6d5ce23b8253c9ef4ea5"
server_key_iv: "b3af8b2e928bd6dbf98edb1b348edbe7"
server_key_version: "encryptionkey@0"
I20260929 16:29:17.029664 10899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:17.035182 10899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:17.036041 10899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:17.036305 10899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:17.037199 10899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:17.038630 10899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:17.038754 10899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:17.038889 10899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:17.038976 10899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:17.072494 10899 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:33079
I20260929 16:29:17.072536 11025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:33079 every 8 connection(s)
I20260929 16:29:17.073887 10899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:17.076639 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 10899
I20260929 16:29:17.076855 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:17.077307 6050 external_mini_cluster.cc:1428] Setting key b45266d777974776c811a879e3c5648f
I20260929 16:29:17.082058 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42781
--builtin_ntp_servers=127.5.232.148:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:29:17.082552 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.076445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:51003 (local address 127.5.232.190:42781)
0929 16:29:17.076681 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.076688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.078293 (+ 1605us) server_negotiation.cc:407] Connection header received
0929 16:29:17.080109 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.080116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.080192 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.080305 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.081741 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.081750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.081758 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.081781 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.081825 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.081843 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.081968 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.082140 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.082205 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:29:17.083849 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.077021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42781 (local address 127.5.232.130:51003)
0929 16:29:17.078139 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:17.078188 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:17.079802 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:17.080542 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:17.080551 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:17.080650 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:17.081424 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:29:17.081428 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:17.081477 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:29:17.081567 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:17.081945 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:17.081952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:17.081958 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:29:17.082213 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 16:29:17.082499 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":417,"threads_started":1}
I20260929 16:29:17.085485 11026 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42781
I20260929 16:29:17.085816 11026 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:17.086452 11026 heartbeater.cc:511] Master 127.5.232.190:42781 requested a full tablet report, sending...
I20260929 16:29:17.087982 10709 ts_manager.cc:194] Registered new tserver with Master: 846bee9149ca49a89738f57b329bd203 (127.5.232.130:33079)
W20260929 16:29:17.418040 11029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:17.418395 11029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:17.418562 11029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:17.425961 11029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:17.426120 11029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:17.426301 11029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:17.435552 11029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:17.436916 11029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:17.438773 11029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:17.452080 11037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:17.452098 11035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:17.452791 11029 server_base.cc:1034] running on GCE node
W20260929 16:29:17.452994 11034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:17.453779 11029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:17.454721 11029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:17.455922 11029 hybrid_clock.cc:648] HybridClock initialized: now 1790699357455882 us; error 30 us; skew 500 ppm
I20260929 16:29:17.456216 11029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:29:17.458717 11029 webserver.cc:460] Webserver started at http://127.5.232.131:33079/ using document root <none> and password file <none>
I20260929 16:29:17.459401 11029 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:17.459491 11029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:17.459766 11029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:17.462778 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "233ee0e915ba46139221822709244fdc"
format_stamp: "Formatted at 2026-09-29 16:29:17 on dist-test-slave-dssx"
server_key: "5e56e355ca0a5791c4a29777c19dff8b"
server_key_iv: "06205cfe78bf1b7b083d0777afd6ece7"
server_key_version: "encryptionkey@0"
I20260929 16:29:17.463590 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "233ee0e915ba46139221822709244fdc"
format_stamp: "Formatted at 2026-09-29 16:29:17 on dist-test-slave-dssx"
server_key: "5e56e355ca0a5791c4a29777c19dff8b"
server_key_iv: "06205cfe78bf1b7b083d0777afd6ece7"
server_key_version: "encryptionkey@0"
I20260929 16:29:17.469278 11029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:29:17.473098 11043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:17.474640 11029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:29:17.474792 11029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "233ee0e915ba46139221822709244fdc"
format_stamp: "Formatted at 2026-09-29 16:29:17 on dist-test-slave-dssx"
server_key: "5e56e355ca0a5791c4a29777c19dff8b"
server_key_iv: "06205cfe78bf1b7b083d0777afd6ece7"
server_key_version: "encryptionkey@0"
I20260929 16:29:17.474995 11029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:17.480850 11029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:17.481832 11029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:17.482059 11029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:17.482949 11029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:17.484334 11029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:17.484421 11029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:17.484496 11029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:17.484544 11029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:17.517258 11029 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:34559
I20260929 16:29:17.517311 11155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:34559 every 8 connection(s)
I20260929 16:29:17.518623 11029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:17.523044 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 11029
I20260929 16:29:17.523252 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:17.523819 6050 external_mini_cluster.cc:1428] Setting key 747cc97fe0207dbbee88bd5debb7d5a1
I20260929 16:29:17.528111 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.521463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:36303 (local address 127.5.232.190:42781)
0929 16:29:17.521635 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.521639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.523309 (+ 1670us) server_negotiation.cc:407] Connection header received
0929 16:29:17.525660 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.525663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.525749 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.525857 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.527331 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.527337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.527342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.527361 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.527398 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.527410 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.527541 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.527772 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.527839 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:29:17.529100 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.521950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42781 (local address 127.5.232.131:36303)
0929 16:29:17.523021 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:17.523086 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:17.525321 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:17.526155 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:17.526165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:17.526271 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:17.526993 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:29:17.526995 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:17.527038 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:29:17.527159 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:17.527508 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:17.527520 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:17.527529 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:29:17.527849 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 16:29:17.528147 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":268,"threads_started":1}
I20260929 16:29:17.531059 11156 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42781
I20260929 16:29:17.531387 11156 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:17.531998 11156 heartbeater.cc:511] Master 127.5.232.190:42781 requested a full tablet report, sending...
I20260929 16:29:17.533701 10709 ts_manager.cc:194] Registered new tserver with Master: 233ee0e915ba46139221822709244fdc (127.5.232.131:34559)
I20260929 16:29:17.539428 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:29:17.546247 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.543819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.5.232.190:42781)
0929 16:29:17.544069 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.544079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.544491 (+ 412us) server_negotiation.cc:407] Connection header received
0929 16:29:17.544626 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.544629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.544682 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.544767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.545670 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.545674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.545677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.545692 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.545721 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.545733 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.545833 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.545872 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.545915 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:29:17.550343 10709 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:17.557472 10709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:17.587504 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.571081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:33079 (local address 127.0.0.1:50884)
0929 16:29:17.583590 (+ 12509us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:17.583616 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:17.583751 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:17.584999 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:17.585007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:17.585117 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:17.586133 (+ 1016us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:29:17.586137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:17.586175 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:29:17.586271 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:17.586748 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:17.586760 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:17.586768 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:29:17.586929 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:29:17.587043 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12288,"thread_start_us":12799,"threads_started":1}
I20260929 16:29:17.588537 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.571455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.5.232.130:33079)
0929 16:29:17.584486 (+ 13031us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.584493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.584523 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:29:17.584591 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.584601 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.584802 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.584986 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.586458 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.586469 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.586486 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.586517 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.586593 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.586622 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.588055 (+ 1433us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.588136 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.588244 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12864,"thread_start_us":12738,"threads_started":1}
I20260929 16:29:17.591535 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.582864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.5.232.129:36667)
0929 16:29:17.588158 (+ 5294us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.588167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.588194 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:29:17.588261 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.588279 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.588538 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.588709 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.590660 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.590673 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.590691 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.590723 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.590788 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.590810 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.590954 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.591082 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.591190 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5121,"thread_start_us":4994,"threads_started":1}
I20260929 16:29:17.591593 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.585688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36667 (local address 127.0.0.1:51034)
0929 16:29:17.586325 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:17.586349 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:17.586474 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:17.589695 (+ 3221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:17.589699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:17.589721 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:17.590411 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:29:17.590414 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:17.590431 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:29:17.590473 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:17.590933 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:17.590939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:17.590944 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:29:17.591093 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 16:29:17.591147 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":762,"threads_started":1}
I20260929 16:29:17.592144 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.570492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.5.232.131:34559)
0929 16:29:17.588686 (+ 18194us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.588705 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.588744 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:29:17.588807 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.588814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.589007 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.589173 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.590879 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.590890 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.590903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.590919 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.590973 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.590993 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.591153 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.591704 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.591795 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18002,"thread_start_us":17850,"threads_started":1}
I20260929 16:29:17.592646 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.570491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:34559 (local address 127.0.0.1:41208)
0929 16:29:17.581030 (+ 10539us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:17.581100 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:17.581277 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:17.589204 (+ 7927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:17.589209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:17.589236 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:17.590040 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:29:17.590046 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:17.590066 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:29:17.590122 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:17.591211 (+ 1089us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:17.591216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:17.591221 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:29:17.591367 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:29:17.591418 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10163,"thread_start_us":10918,"threads_started":1}
I20260929 16:29:17.592864 10961 tablet_service.cc:1467] Processing CreateTablet for tablet 659dab095ecf45f186132f71e9d0dbeb (DEFAULT_TABLE table=test-table [id=112a6497935644f7851759be13c72583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:17.594468 10961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659dab095ecf45f186132f71e9d0dbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:17.595865 11091 tablet_service.cc:1467] Processing CreateTablet for tablet 659dab095ecf45f186132f71e9d0dbeb (DEFAULT_TABLE table=test-table [id=112a6497935644f7851759be13c72583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:17.595716 10830 tablet_service.cc:1467] Processing CreateTablet for tablet 659dab095ecf45f186132f71e9d0dbeb (DEFAULT_TABLE table=test-table [id=112a6497935644f7851759be13c72583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:17.597040 11091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659dab095ecf45f186132f71e9d0dbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:17.597123 10830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659dab095ecf45f186132f71e9d0dbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:17.610025 11174 tablet_bootstrap.cc:492] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: Bootstrap starting.
I20260929 16:29:17.611405 11175 tablet_bootstrap.cc:492] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: Bootstrap starting.
I20260929 16:29:17.613150 11176 tablet_bootstrap.cc:492] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553: Bootstrap starting.
I20260929 16:29:17.615162 11174 tablet_bootstrap.cc:654] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:17.615168 11175 tablet_bootstrap.cc:654] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:17.616106 11176 tablet_bootstrap.cc:654] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:17.616475 11175 log.cc:824] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:17.616475 11174 log.cc:824] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:17.617228 11176 log.cc:824] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:17.620029 11174 tablet_bootstrap.cc:492] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: No bootstrap required, opened a new log
I20260929 16:29:17.620157 11176 tablet_bootstrap.cc:492] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553: No bootstrap required, opened a new log
I20260929 16:29:17.620332 11174 ts_tablet_manager.cc:1397] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260929 16:29:17.620371 11175 tablet_bootstrap.cc:492] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: No bootstrap required, opened a new log
I20260929 16:29:17.620404 11176 ts_tablet_manager.cc:1397] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:29:17.620618 11175 ts_tablet_manager.cc:1397] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 16:29:17.627490 11174 raft_consensus.cc:348] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } }
I20260929 16:29:17.627910 11174 raft_consensus.cc:374] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:17.628001 11174 raft_consensus.cc:729] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233ee0e915ba46139221822709244fdc, State: Initialized, Role: FOLLOWER
I20260929 16:29:17.628682 11174 consensus_queue.cc:260] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } }
I20260929 16:29:17.629823 11156 heartbeater.cc:503] Master 127.5.232.190:42781 was elected leader, sending a full tablet report...
I20260929 16:29:17.630578 11174 ts_tablet_manager.cc:1428] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: Time spent starting tablet: real 0.010s user 0.003s sys 0.008s
I20260929 16:29:17.631326 11176 raft_consensus.cc:348] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } }
I20260929 16:29:17.631326 11175 raft_consensus.cc:348] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } }
I20260929 16:29:17.631789 11176 raft_consensus.cc:374] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:17.631789 11175 raft_consensus.cc:374] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:17.631894 11176 raft_consensus.cc:729] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a164b75037433ab4f2e94652d0d553, State: Initialized, Role: FOLLOWER
I20260929 16:29:17.631894 11175 raft_consensus.cc:729] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846bee9149ca49a89738f57b329bd203, State: Initialized, Role: FOLLOWER
I20260929 16:29:17.632547 11176 consensus_queue.cc:260] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } }
I20260929 16:29:17.632563 11175 consensus_queue.cc:260] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } }
I20260929 16:29:17.638087 11026 heartbeater.cc:503] Master 127.5.232.190:42781 was elected leader, sending a full tablet report...
I20260929 16:29:17.638592 10895 heartbeater.cc:503] Master 127.5.232.190:42781 was elected leader, sending a full tablet report...
I20260929 16:29:17.638912 11175 ts_tablet_manager.cc:1428] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260929 16:29:17.639079 11176 ts_tablet_manager.cc:1428] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553: Time spent starting tablet: real 0.019s user 0.013s sys 0.007s
W20260929 16:29:17.648804 10896 tablet.cc:2369] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:29:17.770357 11157 tablet.cc:2369] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:29:17.775254 11182 raft_consensus.cc:482] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:17.775537 11182 raft_consensus.cc:504] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } }
I20260929 16:29:17.777140 11182 leader_election.cc:288] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 233ee0e915ba46139221822709244fdc (127.5.232.131:34559), 846bee9149ca49a89738f57b329bd203 (127.5.232.130:33079)
I20260929 16:29:17.780514 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.777620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:37805 (local address 127.5.232.130:33079)
0929 16:29:17.777952 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.777961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.778341 (+ 380us) server_negotiation.cc:407] Connection header received
0929 16:29:17.778581 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.778587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.778647 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.778756 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.779838 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.779844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.779848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.779868 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.779900 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.779916 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.780038 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.780138 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.780206 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260929 16:29:17.780642 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.777585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:44157 (local address 127.5.232.131:34559)
0929 16:29:17.777793 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.777801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.778608 (+ 807us) server_negotiation.cc:407] Connection header received
0929 16:29:17.778747 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.778753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.778808 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.778905 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.780043 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.780048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.780052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.780073 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.780122 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.780142 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.780258 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.780353 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.780408 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:29:17.780722 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.777457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:33079 (local address 127.5.232.129:37805)
0929 16:29:17.778212 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:17.778235 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:17.778430 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:17.778818 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:17.778822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:17.778849 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:17.779611 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:29:17.779617 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:17.779636 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:29:17.779688 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:17.780019 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:17.780024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:17.780028 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:29:17.780172 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 16:29:17.780227 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":609,"threads_started":1}
I20260929 16:29:17.780925 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.777540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:34559 (local address 127.5.232.129:44157)
0929 16:29:17.778478 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:17.778516 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:17.778625 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:17.778943 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:17.778946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:17.778966 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:17.779788 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:29:17.779791 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:17.779865 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:29:17.779908 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:17.780233 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:17.780237 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:17.780241 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:29:17.780390 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 16:29:17.780433 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":636,"threads_started":1}
I20260929 16:29:17.781741 10981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659dab095ecf45f186132f71e9d0dbeb" candidate_uuid: "33a164b75037433ab4f2e94652d0d553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846bee9149ca49a89738f57b329bd203" is_pre_election: true
I20260929 16:29:17.781877 11111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659dab095ecf45f186132f71e9d0dbeb" candidate_uuid: "33a164b75037433ab4f2e94652d0d553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233ee0e915ba46139221822709244fdc" is_pre_election: true
I20260929 16:29:17.782328 11111 raft_consensus.cc:2454] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33a164b75037433ab4f2e94652d0d553 in term 0.
I20260929 16:29:17.782328 10981 raft_consensus.cc:2454] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33a164b75037433ab4f2e94652d0d553 in term 0.
I20260929 16:29:17.782963 10784 leader_election.cc:302] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33a164b75037433ab4f2e94652d0d553, 846bee9149ca49a89738f57b329bd203; no voters:
I20260929 16:29:17.783317 11182 raft_consensus.cc:2781] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:17.783479 11182 raft_consensus.cc:482] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:17.783627 11182 raft_consensus.cc:3037] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:17.785833 11182 raft_consensus.cc:504] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } }
I20260929 16:29:17.786500 11182 leader_election.cc:288] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [CANDIDATE]: Term 1 election: Requested vote from peers 233ee0e915ba46139221822709244fdc (127.5.232.131:34559), 846bee9149ca49a89738f57b329bd203 (127.5.232.130:33079)
I20260929 16:29:17.786847 11111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659dab095ecf45f186132f71e9d0dbeb" candidate_uuid: "33a164b75037433ab4f2e94652d0d553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233ee0e915ba46139221822709244fdc"
I20260929 16:29:17.787005 11111 raft_consensus.cc:3037] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:17.787088 10981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659dab095ecf45f186132f71e9d0dbeb" candidate_uuid: "33a164b75037433ab4f2e94652d0d553" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "846bee9149ca49a89738f57b329bd203"
I20260929 16:29:17.787312 10981 raft_consensus.cc:3037] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:17.789047 11111 raft_consensus.cc:2454] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33a164b75037433ab4f2e94652d0d553 in term 1.
I20260929 16:29:17.789443 10785 leader_election.cc:302] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233ee0e915ba46139221822709244fdc, 33a164b75037433ab4f2e94652d0d553; no voters:
I20260929 16:29:17.789680 10981 raft_consensus.cc:2454] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33a164b75037433ab4f2e94652d0d553 in term 1.
I20260929 16:29:17.789737 11182 raft_consensus.cc:2781] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:17.790267 11182 raft_consensus.cc:686] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [term 1 LEADER]: Becoming Leader. State: Replica: 33a164b75037433ab4f2e94652d0d553, State: Running, Role: LEADER
I20260929 16:29:17.790738 11182 consensus_queue.cc:237] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } }
I20260929 16:29:17.795408 10708 catalog_manager.cc:5495] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33a164b75037433ab4f2e94652d0d553 (127.5.232.129). New cstate: current_term: 1 leader_uuid: "33a164b75037433ab4f2e94652d0d553" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33a164b75037433ab4f2e94652d0d553" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 } health_report { overall_health: UNKNOWN } } }
I20260929 16:29:17.815980 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.813990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.5.232.129:36667)
0929 16:29:17.814212 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.814218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.814267 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:29:17.814468 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.814472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.814550 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.814632 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.815416 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.815420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.815423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.815437 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.815465 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.815476 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.815563 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.815638 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.815679 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:29:17.824117 10981 raft_consensus.cc:1261] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [term 1 FOLLOWER]: Refusing update from remote peer 33a164b75037433ab4f2e94652d0d553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:17.824112 11111 raft_consensus.cc:1261] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [term 1 FOLLOWER]: Refusing update from remote peer 33a164b75037433ab4f2e94652d0d553: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:17.825088 11182 consensus_queue.cc:1035] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "233ee0e915ba46139221822709244fdc" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:17.825412 11185 consensus_queue.cc:1035] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553 [LEADER]: Connected to new peer: Peer: permanent_uuid: "846bee9149ca49a89738f57b329bd203" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:29:17.825992 11027 tablet.cc:2369] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:29:17.834240 11188 mvcc.cc:205] Tried to move back new op lower bound from 7334704569637826560 to 7334704569514549248. Current Snapshot: MvccSnapshot[applied={T|T < 7334704569637826560}]
I20260929 16:29:17.836844 11189 mvcc.cc:205] Tried to move back new op lower bound from 7334704569637826560 to 7334704569514549248. Current Snapshot: MvccSnapshot[applied={T|T < 7334704569637826560}]
I20260929 16:29:17.853474 10709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:17.853720 10709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:17.857040 10709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 253560e43dab4bd489ba824ca56f0329: Sending DeleteTablet for 3 replicas of tablet 659dab095ecf45f186132f71e9d0dbeb
I20260929 16:29:17.858220 11091 tablet_service.cc:1514] Processing DeleteTablet for tablet 659dab095ecf45f186132f71e9d0dbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:17 UTC) from {username='slave'} at 127.0.0.1:41208
I20260929 16:29:17.858289 10830 tablet_service.cc:1514] Processing DeleteTablet for tablet 659dab095ecf45f186132f71e9d0dbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:17 UTC) from {username='slave'} at 127.0.0.1:51034
I20260929 16:29:17.858513 10961 tablet_service.cc:1514] Processing DeleteTablet for tablet 659dab095ecf45f186132f71e9d0dbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:17 UTC) from {username='slave'} at 127.0.0.1:50884
I20260929 16:29:17.862653 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 10768
I20260929 16:29:17.868018 11198 tablet_replica.cc:330] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: stopping tablet replica
I20260929 16:29:17.868294 11200 tablet_replica.cc:330] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: stopping tablet replica
I20260929 16:29:17.868515 11198 raft_consensus.cc:2229] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:17.868875 11200 raft_consensus.cc:2229] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:17.869459 11198 raft_consensus.cc:2258] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:17.869604 11200 raft_consensus.cc:2258] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:17.870477 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.863842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.5.232.129:36667)
0929 16:29:17.864059 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.864068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.867646 (+ 3578us) server_negotiation.cc:407] Connection header received
0929 16:29:17.868286 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.868293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.868354 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.868429 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.869703 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.869709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.869713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.869732 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.869770 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.869784 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.869885 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.870121 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.870181 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:29:17.871567 10890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:17.872151 11200 ts_tablet_manager.cc:1905] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:17.872924 11198 ts_tablet_manager.cc:1905] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:17.879331 11200 ts_tablet_manager.cc:1918] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:17.879803 11200 log.cc:1196] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/659dab095ecf45f186132f71e9d0dbeb
I20260929 16:29:17.880070 11198 ts_tablet_manager.cc:1918] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:17.880237 11198 log.cc:1196] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/659dab095ecf45f186132f71e9d0dbeb
I20260929 16:29:17.880574 11200 ts_tablet_manager.cc:1939] T 659dab095ecf45f186132f71e9d0dbeb P 846bee9149ca49a89738f57b329bd203: Deleting consensus metadata
I20260929 16:29:17.880733 11198 ts_tablet_manager.cc:1939] T 659dab095ecf45f186132f71e9d0dbeb P 233ee0e915ba46139221822709244fdc: Deleting consensus metadata
I20260929 16:29:17.882623 10694 catalog_manager.cc:4841] TS 846bee9149ca49a89738f57b329bd203 (127.5.232.130:33079): tablet 659dab095ecf45f186132f71e9d0dbeb (table test-table [id=112a6497935644f7851759be13c72583]) successfully deleted
I20260929 16:29:17.882651 10695 catalog_manager.cc:4841] TS 233ee0e915ba46139221822709244fdc (127.5.232.131:34559): tablet 659dab095ecf45f186132f71e9d0dbeb (table test-table [id=112a6497935644f7851759be13c72583]) successfully deleted
I20260929 16:29:17.970883 11199 tablet_replica.cc:330] T 659dab095ecf45f186132f71e9d0dbeb P 33a164b75037433ab4f2e94652d0d553: stopping tablet replica
I20260929 16:29:17.971220 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 10768
W20260929 16:29:17.990486 10695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.232.129:36667 (error 108)
I20260929 16:29:17.990774 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 10899
I20260929 16:29:17.992609 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.992096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36667 (local address 127.0.0.1:51070)
0929 16:29:17.992284 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:17.992421 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 16:29:17.992864 10695 catalog_manager.cc:4584] TS 33a164b75037433ab4f2e94652d0d553 (127.5.232.129:36667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 659dab095ecf45f186132f71e9d0dbeb: Network error: Client connection negotiation failed: client connection to 127.5.232.129:36667: connect: Connection refused (error 111)
I20260929 16:29:17.993875 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:17.991657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.5.232.130:33079)
0929 16:29:17.991839 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:17.991844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:17.991930 (+ 86us) server_negotiation.cc:407] Connection header received
0929 16:29:17.992067 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:17.992073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:17.992191 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:17.992301 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:17.993319 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:17.993324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:17.993327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:17.993341 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:17.993370 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:17.993430 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:17.993573 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:17.993609 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 16:29:17.993653 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:29:17.994491 11021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:18.046773 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:18.046171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36667 (local address 127.0.0.1:51076)
0929 16:29:18.046412 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:18.046562 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:29:18.047835 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:18.047463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36667 (local address 127.0.0.1:51080)
0929 16:29:18.047617 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:18.047698 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 16:29:18.083562 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 10899
I20260929 16:29:18.100764 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 11029
I20260929 16:29:18.103742 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:18.101601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.5.232.131:34559)
0929 16:29:18.101786 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:18.101792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:18.101812 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:29:18.101955 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:18.101961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:18.102051 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:18.102166 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:18.103065 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:18.103071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:18.103075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:18.103100 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:18.103141 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:18.103153 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:18.103281 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:18.103387 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 16:29:18.103446 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:29:18.104363 11151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:18.108465 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:18.107974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36667 (local address 127.0.0.1:51094)
0929 16:29:18.108145 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:18.108286 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 16:29:18.109614 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:18.109141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36667 (local address 127.0.0.1:51110)
0929 16:29:18.109352 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:18.109455 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 16:29:18.190145 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:18.189655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36667 (local address 127.0.0.1:51122)
0929 16:29:18.189872 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:18.189983 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 16:29:18.191190 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:18.190813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36667 (local address 127.0.0.1:51132)
0929 16:29:18.191008 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:18.191077 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 16:29:18.206724 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 11029
I20260929 16:29:18.223526 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 10678
I20260929 16:29:18.224742 10739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:18.330826 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 10678
2026-09-29T16:29:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2700 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 16:29:18.372025 6050 mini_oidc.cc:152] Starting JWKS server
I20260929 16:29:18.374711 6050 webserver.cc:460] Webserver started at https://127.0.0.1:41341/ using document root <none> and password file <none>
I20260929 16:29:18.374965 6050 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:29:18.375571 6050 webserver.cc:460] Webserver started at http://127.0.0.1:44031/ using document root <none> and password file <none>
2026-09-29T16:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:18Z Disabled control of system clock
I20260929 16:29:18.403062 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40599
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:40599
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41341/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/testchainca.cert with env {}
W20260929 16:29:18.726554 11212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:18.726876 11212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:18.726966 11212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:18.734333 11212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:18.734454 11212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:18.734491 11212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:18.734540 11212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:18.734584 11212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:29:18.734637 11212 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:29:18.734678 11212 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41341/jwks.json
I20260929 16:29:18.744676 11212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:40599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41341/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:18.745967 11212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:18.747200 11212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:18.758836 11217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:18.758801 11218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:18.758881 11220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:18.760082 11212 server_base.cc:1034] running on GCE node
I20260929 16:29:18.761030 11212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:18.762701 11212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:18.763962 11212 hybrid_clock.cc:648] HybridClock initialized: now 1790699358763918 us; error 50 us; skew 500 ppm
I20260929 16:29:18.764364 11212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:29:18.766934 11212 webserver.cc:460] Webserver started at http://127.5.232.190:41203/ using document root <none> and password file <none>
I20260929 16:29:18.767716 11212 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:18.767879 11212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:18.768218 11212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:18.771461 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "53ead1cbbd8048c98cb3fa8a9264fa83"
format_stamp: "Formatted at 2026-09-29 16:29:18 on dist-test-slave-dssx"
server_key: "6a85c7463f0fc3f6241285a61421e95f"
server_key_iv: "1a25833bf05b3fe832250d754cdf3793"
server_key_version: "encryptionkey@0"
I20260929 16:29:18.772336 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "53ead1cbbd8048c98cb3fa8a9264fa83"
format_stamp: "Formatted at 2026-09-29 16:29:18 on dist-test-slave-dssx"
server_key: "6a85c7463f0fc3f6241285a61421e95f"
server_key_iv: "1a25833bf05b3fe832250d754cdf3793"
server_key_version: "encryptionkey@0"
I20260929 16:29:18.778283 11212 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:29:18.782115 11226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:18.783605 11212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:18.783813 11212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "53ead1cbbd8048c98cb3fa8a9264fa83"
format_stamp: "Formatted at 2026-09-29 16:29:18 on dist-test-slave-dssx"
server_key: "6a85c7463f0fc3f6241285a61421e95f"
server_key_iv: "1a25833bf05b3fe832250d754cdf3793"
server_key_version: "encryptionkey@0"
I20260929 16:29:18.784052 11212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:18.801779 11212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:18.802716 11212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:18.803056 11212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:18.821636 11212 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:40599
I20260929 16:29:18.821645 11277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:40599 every 8 connection(s)
I20260929 16:29:18.823055 11212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:18.826949 11278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:18.832392 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 11212
I20260929 16:29:18.832609 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:18.833242 6050 external_mini_cluster.cc:1428] Setting key 40afed6c1525e9dc0e38af8c3e0bc375
I20260929 16:29:18.839079 11278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83: Bootstrap starting.
I20260929 16:29:18.843206 11278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:18.844295 11278 log.cc:824] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:18.847196 11278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83: No bootstrap required, opened a new log
I20260929 16:29:18.849301 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:18.835393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.5.232.190:40599)
0929 16:29:18.836636 (+ 1243us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:18.836653 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:18.836705 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:29:18.839160 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:18.839192 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:18.839801 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:18.840261 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:18.840865 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:18.842896 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:18.844350 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:18.845316 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:18.845592 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:18.846867 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:18.846875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:18.846890 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:18.847528 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:18.847618 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:18.847645 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:18.847772 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:18.848019 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 16:29:18.848335 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":417,"threads_started":1}
I20260929 16:29:18.853335 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:29:18.855780 11278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ead1cbbd8048c98cb3fa8a9264fa83" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40599 } }
I20260929 16:29:18.856184 11278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:18.856302 11278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ead1cbbd8048c98cb3fa8a9264fa83, State: Initialized, Role: FOLLOWER
I20260929 16:29:18.856945 11278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ead1cbbd8048c98cb3fa8a9264fa83" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40599 } }
I20260929 16:29:18.857189 11278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:18.857329 11278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:18.857534 11278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:18.859804 11278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ead1cbbd8048c98cb3fa8a9264fa83" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40599 } }
I20260929 16:29:18.860353 11278 leader_election.cc:302] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53ead1cbbd8048c98cb3fa8a9264fa83; no voters:
I20260929 16:29:18.860920 11278 leader_election.cc:288] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:18.861073 11283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:18.861826 11283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [term 1 LEADER]: Becoming Leader. State: Replica: 53ead1cbbd8048c98cb3fa8a9264fa83, State: Running, Role: LEADER
I20260929 16:29:18.862356 11283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ead1cbbd8048c98cb3fa8a9264fa83" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40599 } }
I20260929 16:29:18.863044 11278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:18.865458 11285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ead1cbbd8048c98cb3fa8a9264fa83. Latest consensus state: current_term: 1 leader_uuid: "53ead1cbbd8048c98cb3fa8a9264fa83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ead1cbbd8048c98cb3fa8a9264fa83" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40599 } } }
I20260929 16:29:18.865409 11284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53ead1cbbd8048c98cb3fa8a9264fa83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ead1cbbd8048c98cb3fa8a9264fa83" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40599 } } }
I20260929 16:29:18.865837 11284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:18.865837 11285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:18.866757 11291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:18.872951 11291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:18.881852 11291 catalog_manager.cc:1347] Generated new cluster ID: 02936cde8c5e4572a336c5ea908ab53c
I20260929 16:29:18.882023 11291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:18.900578 11291 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:18.901962 11291 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:18.913805 11291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53ead1cbbd8048c98cb3fa8a9264fa83: Generated new TSK 0
I20260929 16:29:18.914666 11291 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:29:18.959592 11308 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 16:29:18.959808 11308 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:29:18.960985 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:18.939108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.5.232.190:40599)
0929 16:29:18.939321 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:18.939328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:18.939812 (+ 484us) server_negotiation.cc:407] Connection header received
0929 16:29:18.940117 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:18.940123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:18.940235 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:18.940399 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:29:18.940951 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:18.942135 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:18.943572 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:18.944280 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:18.944417 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:18.944959 (+ 542us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:29:18.960222 (+ 15263us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:29:18.960637 (+ 415us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 16:29:18.961122 11281 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 16:29:18.962545 11307 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:18.938955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40599 (local address 127.0.0.1:41648)
0929 16:29:18.939703 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:18.939721 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:18.939875 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:18.940438 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:18.940442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:18.940452 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:29:18.940774 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:18.940782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:18.942278 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:18.942282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:18.943423 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:18.943434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:18.944456 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:18.944460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:18.944589 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:18.944654 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:29:18.960537 (+ 15883us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:29:18.960901 (+ 364us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.232.190:40599: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":393,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.232.190:40599: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 16:29:18.964517 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 11212
I20260929 16:29:18.965801 11273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:19.058715 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 11212
2026-09-29T16:29:19Z chronyd exiting
I20260929 16:29:19.108701 6050 test_util.cc:175] -----------------------------------------------
I20260929 16:29:19.108829 6050 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699314866137-6050-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (741 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 16:29:19.112526 6050 mini_oidc.cc:152] Starting JWKS server
I20260929 16:29:19.114436 6050 webserver.cc:460] Webserver started at https://127.0.0.1:39979/ using document root <none> and password file <none>
I20260929 16:29:19.114692 6050 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:29:19.115345 6050 webserver.cc:460] Webserver started at http://127.0.0.1:45631/ using document root <none> and password file <none>
2026-09-29T16:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:19Z Disabled control of system clock
I20260929 16:29:19.141805 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:38099
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:38099
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39979/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/testchainca.cert with env {}
W20260929 16:29:19.470954 11315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:19.471336 11315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:19.471491 11315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:19.479410 11315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:19.479570 11315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:19.479661 11315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:19.479759 11315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:19.479835 11315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:29:19.479923 11315 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:29:19.479998 11315 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39979/jwks.json
I20260929 16:29:19.490197 11315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:38099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39979/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:19.491583 11315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:19.492914 11315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:19.504583 11320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:19.504613 11321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:19.504678 11323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:19.505491 11315 server_base.cc:1034] running on GCE node
I20260929 16:29:19.506284 11315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:19.507710 11315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:19.508937 11315 hybrid_clock.cc:648] HybridClock initialized: now 1790699359508881 us; error 56 us; skew 500 ppm
I20260929 16:29:19.509298 11315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:29:19.511595 11315 webserver.cc:460] Webserver started at http://127.5.232.190:36623/ using document root <none> and password file <none>
I20260929 16:29:19.512305 11315 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:19.512442 11315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:19.512766 11315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:19.515743 11315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "7401ec6f52604964af0d9df0623c8cfd"
format_stamp: "Formatted at 2026-09-29 16:29:19 on dist-test-slave-dssx"
server_key: "b3926b77212799faa8d224e746d8b448"
server_key_iv: "899f1cde4753d0399bc8f95a5fdfa37d"
server_key_version: "encryptionkey@0"
I20260929 16:29:19.516567 11315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "7401ec6f52604964af0d9df0623c8cfd"
format_stamp: "Formatted at 2026-09-29 16:29:19 on dist-test-slave-dssx"
server_key: "b3926b77212799faa8d224e746d8b448"
server_key_iv: "899f1cde4753d0399bc8f95a5fdfa37d"
server_key_version: "encryptionkey@0"
I20260929 16:29:19.521768 11315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:29:19.525020 11329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:19.526443 11315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:19.526633 11315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "7401ec6f52604964af0d9df0623c8cfd"
format_stamp: "Formatted at 2026-09-29 16:29:19 on dist-test-slave-dssx"
server_key: "b3926b77212799faa8d224e746d8b448"
server_key_iv: "899f1cde4753d0399bc8f95a5fdfa37d"
server_key_version: "encryptionkey@0"
I20260929 16:29:19.526858 11315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:19.541009 11315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:19.541882 11315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:19.542201 11315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:19.560290 11315 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:38099
I20260929 16:29:19.560310 11380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:38099 every 8 connection(s)
I20260929 16:29:19.561758 11315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:19.565666 11381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:19.571838 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 11315
I20260929 16:29:19.572077 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:19.572907 6050 external_mini_cluster.cc:1428] Setting key 99b8415d0b0db3d082f80ecd6cf29e62
I20260929 16:29:19.577498 11381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd: Bootstrap starting.
I20260929 16:29:19.580687 11381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:19.581840 11381 log.cc:824] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:19.585289 11381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd: No bootstrap required, opened a new log
I20260929 16:29:19.586994 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:19.575000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.5.232.190:38099)
0929 16:29:19.576046 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:19.576058 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:19.576102 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:29:19.577919 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:19.577944 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:19.578433 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:19.578842 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:19.579423 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:19.580691 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:19.582051 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:19.583162 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:19.583332 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:19.584602 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:19.584610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:19.584625 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:19.585176 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:19.585257 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:19.585287 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:19.585432 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:19.585658 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 16:29:19.586033 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":344,"threads_started":1}
I20260929 16:29:19.591008 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:29:19.593611 11381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7401ec6f52604964af0d9df0623c8cfd" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38099 } }
I20260929 16:29:19.594024 11381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:19.594144 11381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7401ec6f52604964af0d9df0623c8cfd, State: Initialized, Role: FOLLOWER
I20260929 16:29:19.594729 11381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7401ec6f52604964af0d9df0623c8cfd" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38099 } }
I20260929 16:29:19.594985 11381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:19.595129 11381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:19.595312 11381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:19.597644 11381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7401ec6f52604964af0d9df0623c8cfd" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38099 } }
I20260929 16:29:19.598212 11381 leader_election.cc:302] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7401ec6f52604964af0d9df0623c8cfd; no voters:
I20260929 16:29:19.598852 11381 leader_election.cc:288] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:19.598984 11386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:19.600843 11381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:19.602466 11386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [term 1 LEADER]: Becoming Leader. State: Replica: 7401ec6f52604964af0d9df0623c8cfd, State: Running, Role: LEADER
I20260929 16:29:19.603040 11386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7401ec6f52604964af0d9df0623c8cfd" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38099 } }
I20260929 16:29:19.606715 11387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7401ec6f52604964af0d9df0623c8cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7401ec6f52604964af0d9df0623c8cfd" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38099 } } }
I20260929 16:29:19.607155 11387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:19.609573 11388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7401ec6f52604964af0d9df0623c8cfd. Latest consensus state: current_term: 1 leader_uuid: "7401ec6f52604964af0d9df0623c8cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7401ec6f52604964af0d9df0623c8cfd" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38099 } } }
I20260929 16:29:19.610312 11388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd [sys.catalog]: This master's current role is: LEADER
W20260929 16:29:19.613598 11400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:29:19.613744 11400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:29:19.613895 11401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:19.615559 11401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:19.625044 11401 catalog_manager.cc:1347] Generated new cluster ID: e9e62ef58328439f900c109cbd741a04
I20260929 16:29:19.625285 11401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:19.643391 11401 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:19.645504 11401 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:19.659796 11401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7401ec6f52604964af0d9df0623c8cfd: Generated new TSK 0
I20260929 16:29:19.660763 11401 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:29:19.701442 11411 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 16:29:19.701601 11411 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:29:19.702721 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:19.682072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.5.232.190:38099)
0929 16:29:19.682316 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:19.682327 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:19.682793 (+ 466us) server_negotiation.cc:407] Connection header received
0929 16:29:19.683044 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:19.683049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:19.683185 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:19.683406 (+ 221us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:29:19.683911 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:19.684750 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:19.686219 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:19.686722 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:19.686879 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:19.687397 (+ 518us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:29:19.702026 (+ 14629us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:29:19.702444 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55446: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 16:29:19.702857 11384 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55446: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 16:29:19.702857 11410 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:19.682019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:38099 (local address 127.0.0.1:55446)
0929 16:29:19.682664 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:19.682687 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:19.682866 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:19.683443 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:19.683449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:19.683464 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:29:19.683744 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:19.683754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:19.684903 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:19.684909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:19.686061 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:19.686073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:19.686910 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:19.686915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:19.687073 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:19.687109 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:29:19.702302 (+ 15193us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:29:19.702517 (+ 215us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.232.190:38099: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":357,"threads_started":1}
I20260929 16:29:19.704613 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 11315
I20260929 16:29:19.705883 11376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:19.807540 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 11315
2026-09-29T16:29:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (750 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 16:29:19.863078 6050 mini_oidc.cc:152] Starting JWKS server
I20260929 16:29:19.865058 6050 webserver.cc:460] Webserver started at https://127.0.0.1:37947/ using document root <none> and password file <none>
I20260929 16:29:19.865305 6050 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:29:19.866055 6050 webserver.cc:460] Webserver started at http://127.0.0.1:40233/ using document root <none> and password file <none>
2026-09-29T16:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:19Z Disabled control of system clock
I20260929 16:29:19.894863 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:39179
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:39179
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37947/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:29:20.228157 11418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:20.228485 11418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:20.228572 11418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:20.237111 11418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:20.237319 11418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:20.237434 11418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:20.237543 11418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:20.237600 11418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:29:20.237711 11418 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:29:20.237790 11418 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37947/jwks.json
I20260929 16:29:20.248857 11418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:39179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:39179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37947/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:20.250202 11418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:20.252082 11418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:20.265781 11423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:20.265720 11424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:20.265946 11426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:20.266661 11418 server_base.cc:1034] running on GCE node
I20260929 16:29:20.267545 11418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:20.269119 11418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:20.270367 11418 hybrid_clock.cc:648] HybridClock initialized: now 1790699360270277 us; error 86 us; skew 500 ppm
I20260929 16:29:20.270753 11418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:29:20.273181 11418 webserver.cc:460] Webserver started at http://127.5.232.190:44445/ using document root <none> and password file <none>
I20260929 16:29:20.274108 11418 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:20.274248 11418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:20.274590 11418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:20.277719 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "c26d94a29da74503b36492d5badb2ab7"
format_stamp: "Formatted at 2026-09-29 16:29:20 on dist-test-slave-dssx"
server_key: "10b69cf760ad03fcb3ed7d9f523c43d1"
server_key_iv: "bf42fc32a49e892e60441b1c68216d0d"
server_key_version: "encryptionkey@0"
I20260929 16:29:20.278573 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "c26d94a29da74503b36492d5badb2ab7"
format_stamp: "Formatted at 2026-09-29 16:29:20 on dist-test-slave-dssx"
server_key: "10b69cf760ad03fcb3ed7d9f523c43d1"
server_key_iv: "bf42fc32a49e892e60441b1c68216d0d"
server_key_version: "encryptionkey@0"
I20260929 16:29:20.284291 11418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:29:20.288010 11432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:20.289649 11418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:20.289861 11418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "c26d94a29da74503b36492d5badb2ab7"
format_stamp: "Formatted at 2026-09-29 16:29:20 on dist-test-slave-dssx"
server_key: "10b69cf760ad03fcb3ed7d9f523c43d1"
server_key_iv: "bf42fc32a49e892e60441b1c68216d0d"
server_key_version: "encryptionkey@0"
I20260929 16:29:20.290102 11418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:20.333514 11418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:20.334425 11418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:20.334760 11418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:20.353927 11418 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:39179
I20260929 16:29:20.353948 11483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:39179 every 8 connection(s)
I20260929 16:29:20.355379 11418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:20.359304 11484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:20.365897 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 11418
I20260929 16:29:20.366117 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:20.366825 6050 external_mini_cluster.cc:1428] Setting key 3a9cb6dd4a8729d699c757b5781669fb
I20260929 16:29:20.370658 11484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7: Bootstrap starting.
I20260929 16:29:20.374300 11484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:20.375672 11484 log.cc:824] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:20.379282 11484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7: No bootstrap required, opened a new log
I20260929 16:29:20.381655 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:20.368865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.5.232.190:39179)
0929 16:29:20.369951 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:20.369963 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:20.370006 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:29:20.371792 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:20.371817 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:20.372346 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:20.372712 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:20.373358 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:20.374831 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:20.376752 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:20.377683 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:20.377835 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:20.379087 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:20.379094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:20.379109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:20.379749 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:20.379829 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:20.379854 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:20.379935 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:20.380205 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:29:20.380566 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":381,"threads_started":1}
I20260929 16:29:20.385732 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:29:20.388525 11484 raft_consensus.cc:348] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26d94a29da74503b36492d5badb2ab7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 39179 } }
I20260929 16:29:20.389009 11484 raft_consensus.cc:374] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:20.389140 11484 raft_consensus.cc:729] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26d94a29da74503b36492d5badb2ab7, State: Initialized, Role: FOLLOWER
I20260929 16:29:20.389878 11484 consensus_queue.cc:260] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26d94a29da74503b36492d5badb2ab7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 39179 } }
I20260929 16:29:20.390185 11484 raft_consensus.cc:388] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:20.390341 11484 raft_consensus.cc:482] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:20.390527 11484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:20.392846 11484 raft_consensus.cc:504] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26d94a29da74503b36492d5badb2ab7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 39179 } }
I20260929 16:29:20.393473 11484 leader_election.cc:302] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c26d94a29da74503b36492d5badb2ab7; no voters:
I20260929 16:29:20.394174 11484 leader_election.cc:288] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:20.394320 11489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:20.395287 11489 raft_consensus.cc:686] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [term 1 LEADER]: Becoming Leader. State: Replica: c26d94a29da74503b36492d5badb2ab7, State: Running, Role: LEADER
I20260929 16:29:20.395888 11489 consensus_queue.cc:237] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26d94a29da74503b36492d5badb2ab7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 39179 } }
I20260929 16:29:20.396785 11484 sys_catalog.cc:564] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:20.398931 11490 sys_catalog.cc:455] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c26d94a29da74503b36492d5badb2ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26d94a29da74503b36492d5badb2ab7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 39179 } } }
I20260929 16:29:20.399003 11491 sys_catalog.cc:455] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c26d94a29da74503b36492d5badb2ab7. Latest consensus state: current_term: 1 leader_uuid: "c26d94a29da74503b36492d5badb2ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c26d94a29da74503b36492d5badb2ab7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 39179 } } }
I20260929 16:29:20.399327 11490 sys_catalog.cc:458] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:20.399410 11491 sys_catalog.cc:458] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:20.401686 11495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:20.407635 11495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:20.417155 11495 catalog_manager.cc:1347] Generated new cluster ID: a69e8c9add584e659834171d0443121b
I20260929 16:29:20.417289 11495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:20.431324 11495 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:20.432643 11495 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:20.446239 11495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c26d94a29da74503b36492d5badb2ab7: Generated new TSK 0
I20260929 16:29:20.447438 11495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:29:20.483078 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:20.474143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.5.232.190:39179)
0929 16:29:20.474360 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:20.474367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:20.474829 (+ 462us) server_negotiation.cc:407] Connection header received
0929 16:29:20.475066 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:20.475072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:20.475168 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:20.475357 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:29:20.475788 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:20.476964 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:20.478509 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:20.479209 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:20.479339 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:20.479850 (+ 511us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:29:20.482379 (+ 2529us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:29:20.482734 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52676: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 16:29:20.483229 11487 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52676: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 16:29:20.484786 11513 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:20.473998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:39179 (local address 127.0.0.1:52676)
0929 16:29:20.474719 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:20.474736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:20.474869 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:20.475373 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:20.475376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:20.475385 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:29:20.475622 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:20.475629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:20.477149 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:20.477153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:20.478354 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:20.478368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:20.479406 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:20.479411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:20.479539 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:20.479574 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:29:20.482626 (+ 3052us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:29:20.484370 (+ 1744us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.232.190:39179: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37947/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":1409,"thread_start_us":405,"threads_started":1}
I20260929 16:29:20.486687 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 11418
I20260929 16:29:20.487926 11479 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 16:29:20.494777 11514 webserver.cc:533] Webserver: sslize failed:
E20260929 16:29:20.494880 11514 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:29:20.581451 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 11418
2026-09-29T16:29:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (777 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T16:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:20Z Disabled control of system clock
I20260929 16:29:20.668982 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:36623
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:36623
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/testchainca.cert with env {}
W20260929 16:29:21.000792 11519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:21.001101 11519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:21.001199 11519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:21.008677 11519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:21.008791 11519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:21.008831 11519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:21.008885 11519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:21.008930 11519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:29:21.009001 11519 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:29:21.009039 11519 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 16:29:21.019153 11519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:36623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:36623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:21.020622 11519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:21.022035 11519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:21.035092 11524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:21.035138 11525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:21.035192 11527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:21.036314 11519 server_base.cc:1034] running on GCE node
I20260929 16:29:21.037189 11519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:21.038762 11519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:21.040036 11519 hybrid_clock.cc:648] HybridClock initialized: now 1790699361039971 us; error 59 us; skew 500 ppm
I20260929 16:29:21.040442 11519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:29:21.042857 11519 webserver.cc:460] Webserver started at http://127.5.232.190:36279/ using document root <none> and password file <none>
I20260929 16:29:21.043613 11519 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:21.043784 11519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:21.044126 11519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:21.047197 11519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "7e2f6ab4329145c9896108601a944506"
format_stamp: "Formatted at 2026-09-29 16:29:21 on dist-test-slave-dssx"
server_key: "26b6b718e073bdb20e3f0e06ce1839fb"
server_key_iv: "f025682079b4150f2941da232157e62b"
server_key_version: "encryptionkey@0"
I20260929 16:29:21.048071 11519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "7e2f6ab4329145c9896108601a944506"
format_stamp: "Formatted at 2026-09-29 16:29:21 on dist-test-slave-dssx"
server_key: "26b6b718e073bdb20e3f0e06ce1839fb"
server_key_iv: "f025682079b4150f2941da232157e62b"
server_key_version: "encryptionkey@0"
I20260929 16:29:21.053606 11519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:29:21.057152 11533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:21.058828 11519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:21.058997 11519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "7e2f6ab4329145c9896108601a944506"
format_stamp: "Formatted at 2026-09-29 16:29:21 on dist-test-slave-dssx"
server_key: "26b6b718e073bdb20e3f0e06ce1839fb"
server_key_iv: "f025682079b4150f2941da232157e62b"
server_key_version: "encryptionkey@0"
I20260929 16:29:21.059190 11519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:21.078100 11519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:21.078948 11519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:21.079210 11519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:21.097252 11519 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:36623
I20260929 16:29:21.097275 11584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:36623 every 8 connection(s)
I20260929 16:29:21.098748 11519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:21.102717 11585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:21.108945 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 11519
I20260929 16:29:21.109179 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:21.110086 6050 external_mini_cluster.cc:1428] Setting key 0c9c9d32ca5997982415242ce43213d1
I20260929 16:29:21.116158 11585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506: Bootstrap starting.
I20260929 16:29:21.120175 11585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:21.121332 11585 log.cc:824] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:21.124631 11585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506: No bootstrap required, opened a new log
I20260929 16:29:21.130637 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:21.112151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.5.232.190:36623)
0929 16:29:21.113358 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:21.113399 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:21.113457 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:29:21.115818 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:21.115861 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:21.116522 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:21.116994 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:21.117569 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:21.119589 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:21.121063 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:21.122245 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:21.122397 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:21.127488 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:21.127500 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:21.127521 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:29:21.128326 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:21.128441 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:29:21.128471 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:21.128608 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:21.128809 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 16:29:21.129192 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":357,"threads_started":1}
I20260929 16:29:21.133914 11585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2f6ab4329145c9896108601a944506" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36623 } }
I20260929 16:29:21.134292 11585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:21.134404 11585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2f6ab4329145c9896108601a944506, State: Initialized, Role: FOLLOWER
I20260929 16:29:21.134788 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:29:21.135068 11585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2f6ab4329145c9896108601a944506" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36623 } }
I20260929 16:29:21.135317 11585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:21.135469 11585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:21.135646 11585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:21.138093 11585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2f6ab4329145c9896108601a944506" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36623 } }
I20260929 16:29:21.138654 11585 leader_election.cc:302] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e2f6ab4329145c9896108601a944506; no voters:
I20260929 16:29:21.139187 11585 leader_election.cc:288] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:21.139365 11590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:21.140029 11590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [term 1 LEADER]: Becoming Leader. State: Replica: 7e2f6ab4329145c9896108601a944506, State: Running, Role: LEADER
I20260929 16:29:21.140551 11590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2f6ab4329145c9896108601a944506" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36623 } }
I20260929 16:29:21.141358 11585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:21.143604 11591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e2f6ab4329145c9896108601a944506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2f6ab4329145c9896108601a944506" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36623 } } }
I20260929 16:29:21.144421 11591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:21.144204 11592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e2f6ab4329145c9896108601a944506. Latest consensus state: current_term: 1 leader_uuid: "7e2f6ab4329145c9896108601a944506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2f6ab4329145c9896108601a944506" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36623 } } }
I20260929 16:29:21.144546 11592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:21.146620 11597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:21.151576 11597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:21.160503 11597 catalog_manager.cc:1347] Generated new cluster ID: 656a64c691634dfd8d980d2b57bdd2d3
I20260929 16:29:21.160642 11597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:21.169708 11597 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:21.170997 11597 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:21.181006 11597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e2f6ab4329145c9896108601a944506: Generated new TSK 0
I20260929 16:29:21.181962 11597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:29:21.276865 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:21.203668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.5.232.190:36623)
0929 16:29:21.203913 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:21.203920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:21.204259 (+ 339us) server_negotiation.cc:407] Connection header received
0929 16:29:21.204508 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:21.204512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:21.204589 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:21.204713 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:29:21.205299 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:21.206079 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:21.208143 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:21.208568 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:21.208657 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:21.209232 (+ 575us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:29:21.276275 (+ 67043us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:29:21.276610 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34380: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":107}
W20260929 16:29:21.277029 11588 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34380: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 16:29:21.277321 11614 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:21.203453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:36623 (local address 127.0.0.1:34380)
0929 16:29:21.204146 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:21.204166 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:21.204322 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:21.204766 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:21.204772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:21.204786 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:29:21.205143 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:21.205157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:21.206233 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:21.206239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:21.207990 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:21.208006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:21.208743 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:21.208749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:21.208934 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:21.208982 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:29:21.276580 (+ 67598us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:29:21.276927 (+ 347us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.232.190:36623: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":354,"threads_started":1}
I20260929 16:29:21.278861 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 11519
I20260929 16:29:21.279994 11580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:21.377830 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 11519
2026-09-29T16:29:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (771 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (412 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1485 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:23 dist-test-slave-dssx krb5kdc[11630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:23 dist-test-slave-dssx krb5kdc[11630](info): set up 2 sockets
Sep 29 16:29:23 dist-test-slave-dssx krb5kdc[11630](info): commencing operation
krb5kdc: starting...
W20260929 16:29:24.354065 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:29:24 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:29:24.561542 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:44351
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:24.889421 11643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:24.889792 11643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:24.889993 11643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:24.897190 11643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:24.897312 11643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:24.897431 11643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:24.897537 11643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:24.897617 11643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:24.907724 11643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:24.909091 11643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:24.910487 11643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:24.922503 11648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:24.922536 11651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:24.922576 11649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:24.923379 11643 server_base.cc:1034] running on GCE node
I20260929 16:29:24.924197 11643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:24.925695 11643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:24.926930 11643 hybrid_clock.cc:648] HybridClock initialized: now 1790699364926833 us; error 63 us; skew 500 ppm
I20260929 16:29:24.927327 11643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:24 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:24.930689 11643 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:29:24.932158 11643 webserver.cc:460] Webserver started at http://127.5.232.190:39031/ using document root <none> and password file <none>
I20260929 16:29:24.932819 11643 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:24.932931 11643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:24.933185 11643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:24.936231 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "51c58773866f419598527458eb976561"
format_stamp: "Formatted at 2026-09-29 16:29:24 on dist-test-slave-dssx"
server_key: "a437fe8bb441682f395b4a9c0f92a738"
server_key_iv: "80066c713f0f11c974a34517d18e5324"
server_key_version: "encryptionkey@0"
I20260929 16:29:24.937018 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "51c58773866f419598527458eb976561"
format_stamp: "Formatted at 2026-09-29 16:29:24 on dist-test-slave-dssx"
server_key: "a437fe8bb441682f395b4a9c0f92a738"
server_key_iv: "80066c713f0f11c974a34517d18e5324"
server_key_version: "encryptionkey@0"
I20260929 16:29:24.942633 11643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 16:29:24.946336 11658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:24.947786 11643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:24.947994 11643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "51c58773866f419598527458eb976561"
format_stamp: "Formatted at 2026-09-29 16:29:24 on dist-test-slave-dssx"
server_key: "a437fe8bb441682f395b4a9c0f92a738"
server_key_iv: "80066c713f0f11c974a34517d18e5324"
server_key_version: "encryptionkey@0"
I20260929 16:29:24.948206 11643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:24.969476 11643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:24.974229 11643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:24.974532 11643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:24.992638 11643 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:44351
I20260929 16:29:24.992663 11709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:44351 every 8 connection(s)
I20260929 16:29:24.994199 11643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:24.997928 11710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:25.002131 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 11643
I20260929 16:29:25.002390 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:25.002980 6050 external_mini_cluster.cc:1428] Setting key 8e1dd4a19e6b4205137160b625b88d12
I20260929 16:29:25.008853 11710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561: Bootstrap starting.
I20260929 16:29:25.012069 11710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:25.013068 11710 log.cc:824] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:25.016533 11710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561: No bootstrap required, opened a new log
Sep 29 16:29:25 dist-test-slave-dssx krb5kdc[11630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:25.027311 11713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:25.005036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.5.232.190:44351)
0929 16:29:25.005846 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:25.005858 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:25.005903 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:29:25.007654 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:25.007678 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:25.008091 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:25.008468 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:25.010020 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.011393 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.012675 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.013457 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.013641 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:25.018180 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:25.018190 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:25.018209 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:25.018259 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:25.022380 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:25.023076 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:25.023087 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:25.023096 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:25.023321 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:25.023771 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:25.023777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:25.023781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:25.024593 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:25.024888 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:25.025255 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 16:29:25.025805 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":370,"threads_started":1}
I20260929 16:29:25.027774 11710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c58773866f419598527458eb976561" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44351 } }
I20260929 16:29:25.028162 11710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:25.028275 11710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c58773866f419598527458eb976561, State: Initialized, Role: FOLLOWER
I20260929 16:29:25.028846 11710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c58773866f419598527458eb976561" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44351 } }
I20260929 16:29:25.029122 11710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:25.029271 11710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:25.029484 11710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:25.031728 11710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c58773866f419598527458eb976561" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44351 } }
I20260929 16:29:25.032218 11710 leader_election.cc:302] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51c58773866f419598527458eb976561; no voters:
I20260929 16:29:25.032773 11710 leader_election.cc:288] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:25.032929 11715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:25.034268 11715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [term 1 LEADER]: Becoming Leader. State: Replica: 51c58773866f419598527458eb976561, State: Running, Role: LEADER
I20260929 16:29:25.034763 11715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c58773866f419598527458eb976561" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44351 } }
I20260929 16:29:25.034964 11710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:25.037778 11717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51c58773866f419598527458eb976561. Latest consensus state: current_term: 1 leader_uuid: "51c58773866f419598527458eb976561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c58773866f419598527458eb976561" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44351 } } }
I20260929 16:29:25.037978 11716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51c58773866f419598527458eb976561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c58773866f419598527458eb976561" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44351 } } }
I20260929 16:29:25.038256 11717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:25.038277 11716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:25.040359 11724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:25.045866 11724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:25.055639 11724 catalog_manager.cc:1347] Generated new cluster ID: 653d6ff860b5491e8550b8386606f274
I20260929 16:29:25.055764 11724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:25.072829 11724 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:25.074236 11724 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:25.080451 11724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51c58773866f419598527458eb976561: Generated new TSK 0
I20260929 16:29:25.081521 11724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:29:25.144933 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44351
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:25.493825 11736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:25.494129 11736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:25.494297 11736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:25.501681 11736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:25.501829 11736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:25.502010 11736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:25.512370 11736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:25.513734 11736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:25.515234 11736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:25.529318 11741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:25.529364 11742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:25.529314 11744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:25.530411 11736 server_base.cc:1034] running on GCE node
I20260929 16:29:25.531054 11736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:25.531996 11736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:25.533216 11736 hybrid_clock.cc:648] HybridClock initialized: now 1790699365533167 us; error 44 us; skew 500 ppm
I20260929 16:29:25.533583 11736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:25 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:25.536971 11736 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:29:25.538442 11736 webserver.cc:460] Webserver started at http://127.5.232.129:39661/ using document root <none> and password file <none>
I20260929 16:29:25.539187 11736 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:25.539323 11736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:25.539667 11736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:25.542672 11736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "c39591472c98436c8a8646676fe605c1"
format_stamp: "Formatted at 2026-09-29 16:29:25 on dist-test-slave-dssx"
server_key: "b99758fa7a6d17f08e57d9b648ac0f2c"
server_key_iv: "fb3ff4c66e8c8eb0ab8dd3fecf097e8e"
server_key_version: "encryptionkey@0"
I20260929 16:29:25.543452 11736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "c39591472c98436c8a8646676fe605c1"
format_stamp: "Formatted at 2026-09-29 16:29:25 on dist-test-slave-dssx"
server_key: "b99758fa7a6d17f08e57d9b648ac0f2c"
server_key_iv: "fb3ff4c66e8c8eb0ab8dd3fecf097e8e"
server_key_version: "encryptionkey@0"
I20260929 16:29:25.548452 11736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:29:25.551771 11751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:25.553215 11736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:25.553437 11736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "c39591472c98436c8a8646676fe605c1"
format_stamp: "Formatted at 2026-09-29 16:29:25 on dist-test-slave-dssx"
server_key: "b99758fa7a6d17f08e57d9b648ac0f2c"
server_key_iv: "fb3ff4c66e8c8eb0ab8dd3fecf097e8e"
server_key_version: "encryptionkey@0"
I20260929 16:29:25.553653 11736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:25.574349 11736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:25.578383 11736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:25.578672 11736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:25.579675 11736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:25.580963 11736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:25.581074 11736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:25.581209 11736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:25.581277 11736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:25.612891 11736 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:43393
I20260929 16:29:25.612938 11863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:43393 every 8 connection(s)
I20260929 16:29:25.614282 11736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:25.618547 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 11736
I20260929 16:29:25.618801 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:25.619410 6050 external_mini_cluster.cc:1428] Setting key 93bd72d050473ddaa47df39c62862506
Sep 29 16:29:25 dist-test-slave-dssx krb5kdc[11630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:25.636212 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:25.616859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:60923 (local address 127.5.232.190:44351)
0929 16:29:25.617286 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:25.617291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:25.618376 (+ 1085us) server_negotiation.cc:407] Connection header received
0929 16:29:25.620364 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:25.620370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:25.620458 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:25.620571 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:25.622231 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.622936 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.624498 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.624901 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.625044 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:25.629141 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:25.629151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:25.629159 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:25.629206 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:25.632704 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:25.633523 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:25.633532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:25.633538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:25.633684 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:25.634104 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:25.634112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:25.634118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:25.634510 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:25.634686 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:25.635466 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 16:29:25.635745 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":212,"threads_started":1}
I20260929 16:29:25.637013 11867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:25.617507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44351 (local address 127.5.232.129:60923)
0929 16:29:25.618228 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:25.618265 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:25.620085 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:25.620813 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:25.620821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:25.621501 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:25.622069 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:25.622088 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.623099 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.623103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:25.624357 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:25.624369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.625137 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.625145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:25.625440 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:25.626206 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:25.626248 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:25.628901 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:25.632866 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:25.632873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:25.632888 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:25.633345 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:25.633790 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:25.633793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:25.633795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:25.633981 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:25.634657 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:25.634666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:25.635019 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 16:29:25.635294 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 16:29:25.635772 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":275,"threads_started":1}
I20260929 16:29:25.638650 11864 heartbeater.cc:347] Connected to a master server at 127.5.232.190:44351
I20260929 16:29:25.638947 11864 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:25.639680 11864 heartbeater.cc:511] Master 127.5.232.190:44351 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
I20260929 16:29:25.641772 11675 ts_manager.cc:194] Registered new tserver with Master: c39591472c98436c8a8646676fe605c1 (127.5.232.129:43393)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
I20260929 16:29:25.643810 11675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:60923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:29:25.666283 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44351
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:26.009599 11870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:26.009995 11870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:26.010149 11870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:26.019937 11870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:26.020069 11870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:26.020183 11870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:26.030169 11870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:26.031422 11870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:26.032670 11870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:26.045975 11876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:26.046017 11875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:26.046060 11878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:26.047132 11870 server_base.cc:1034] running on GCE node
I20260929 16:29:26.047719 11870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:26.048650 11870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:26.049847 11870 hybrid_clock.cc:648] HybridClock initialized: now 1790699366049783 us; error 55 us; skew 500 ppm
I20260929 16:29:26.050243 11870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.053747 11870 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:29:26.055200 11870 webserver.cc:460] Webserver started at http://127.5.232.130:34421/ using document root <none> and password file <none>
I20260929 16:29:26.055868 11870 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:26.055967 11870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:26.056236 11870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:26.059458 11870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "43ca85db1cb14d91868715157be353b2"
format_stamp: "Formatted at 2026-09-29 16:29:26 on dist-test-slave-dssx"
server_key: "78b37947a14987fc612e75b2e23ba902"
server_key_iv: "f3cfc5cb088e950e9ae20008b7f7f020"
server_key_version: "encryptionkey@0"
I20260929 16:29:26.060231 11870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "43ca85db1cb14d91868715157be353b2"
format_stamp: "Formatted at 2026-09-29 16:29:26 on dist-test-slave-dssx"
server_key: "78b37947a14987fc612e75b2e23ba902"
server_key_iv: "f3cfc5cb088e950e9ae20008b7f7f020"
server_key_version: "encryptionkey@0"
I20260929 16:29:26.065735 11870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:29:26.069245 11885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.070720 11870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:26.070865 11870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "43ca85db1cb14d91868715157be353b2"
format_stamp: "Formatted at 2026-09-29 16:29:26 on dist-test-slave-dssx"
server_key: "78b37947a14987fc612e75b2e23ba902"
server_key_iv: "f3cfc5cb088e950e9ae20008b7f7f020"
server_key_version: "encryptionkey@0"
I20260929 16:29:26.071069 11870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:26.088454 11870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:26.092914 11870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:26.093182 11870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:26.094259 11870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:26.095584 11870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:26.095660 11870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.095779 11870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:26.095847 11870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.128489 11870 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:39897
I20260929 16:29:26.128549 11997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:39897 every 8 connection(s)
I20260929 16:29:26.129886 11870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:26.136827 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 11870
I20260929 16:29:26.137032 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:26.137506 6050 external_mini_cluster.cc:1428] Setting key 5299536d8b63add64b045f98c8118328
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:26.157438 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.132500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:43025 (local address 127.5.232.190:44351)
0929 16:29:26.132697 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.132704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.134023 (+ 1319us) server_negotiation.cc:407] Connection header received
0929 16:29:26.135857 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.135862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.135959 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.136079 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.137781 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.138687 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.140480 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.140904 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.141050 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.149952 (+ 8902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.149963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.149971 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.150022 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.153846 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.154791 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.154800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.154807 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.154974 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.155401 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.155410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.155416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.155838 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.156023 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.156723 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.156979 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:29:26.158289 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.133082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44351 (local address 127.5.232.130:43025)
0929 16:29:26.133864 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:26.133908 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:26.135610 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:26.136312 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:26.136321 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:26.136839 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:26.137583 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.137609 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.138837 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.138841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.140336 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.140349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.141145 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.141152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.141434 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.146633 (+ 5199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:26.146680 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:26.149633 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:26.154021 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:26.154030 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:26.154045 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:26.154625 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:26.155096 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:26.155099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:26.155101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:26.155279 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:26.155963 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:26.155969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:26.156327 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 16:29:26.156570 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 16:29:26.156972 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
I20260929 16:29:26.159806 11998 heartbeater.cc:347] Connected to a master server at 127.5.232.190:44351
I20260929 16:29:26.160157 11998 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
I20260929 16:29:26.160997 11998 heartbeater.cc:511] Master 127.5.232.190:44351 requested a full tablet report, sending...
I20260929 16:29:26.162518 11675 ts_manager.cc:194] Registered new tserver with Master: 43ca85db1cb14d91868715157be353b2 (127.5.232.130:39897)
I20260929 16:29:26.163311 11675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:43025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:29:26.185024 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44351
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:26.519155 12003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:26.519446 12003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:26.519541 12003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:26.527129 12003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:26.527249 12003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:26.527412 12003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:26.537747 12003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:26.539132 12003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:26.540493 12003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:26.553931 12009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:26.553979 12011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:26.553890 12008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:26.559988 12003 server_base.cc:1034] running on GCE node
I20260929 16:29:26.560642 12003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:26.561731 12003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:26.562963 12003 hybrid_clock.cc:648] HybridClock initialized: now 1790699366562881 us; error 75 us; skew 500 ppm
I20260929 16:29:26.563306 12003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.566648 12003 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:29:26.568259 12003 webserver.cc:460] Webserver started at http://127.5.232.131:39113/ using document root <none> and password file <none>
I20260929 16:29:26.569003 12003 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:26.569144 12003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:26.569537 12003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:26.572552 12003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "a38da341d56b487f8d0138a6c3054e90"
format_stamp: "Formatted at 2026-09-29 16:29:26 on dist-test-slave-dssx"
server_key: "fe60f90af7916b4dbce5e469ea0ce6b6"
server_key_iv: "cac6063a39e4687bf71ef580a8485f5c"
server_key_version: "encryptionkey@0"
I20260929 16:29:26.573361 12003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "a38da341d56b487f8d0138a6c3054e90"
format_stamp: "Formatted at 2026-09-29 16:29:26 on dist-test-slave-dssx"
server_key: "fe60f90af7916b4dbce5e469ea0ce6b6"
server_key_iv: "cac6063a39e4687bf71ef580a8485f5c"
server_key_version: "encryptionkey@0"
I20260929 16:29:26.578738 12003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:29:26.582177 12018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.583531 12003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:26.583705 12003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "a38da341d56b487f8d0138a6c3054e90"
format_stamp: "Formatted at 2026-09-29 16:29:26 on dist-test-slave-dssx"
server_key: "fe60f90af7916b4dbce5e469ea0ce6b6"
server_key_iv: "cac6063a39e4687bf71ef580a8485f5c"
server_key_version: "encryptionkey@0"
I20260929 16:29:26.583912 12003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:26.606062 12003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:26.610385 12003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:26.610690 12003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:26.611670 12003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:26.613149 12003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:26.613252 12003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.613426 12003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:26.613502 12003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 16:29:26.646867 12003 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:33483
I20260929 16:29:26.646915 12130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:33483 every 8 connection(s)
I20260929 16:29:26.647775 11864 heartbeater.cc:503] Master 127.5.232.190:44351 was elected leader, sending a full tablet report...
I20260929 16:29:26.648402 12003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:26.656119 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 12003
I20260929 16:29:26.656344 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:26.656803 6050 external_mini_cluster.cc:1428] Setting key d44ad320ddbb416796cfce43c026cc9c
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:26.669759 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.651394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:38743 (local address 127.5.232.190:44351)
0929 16:29:26.651611 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.651615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.652884 (+ 1269us) server_negotiation.cc:407] Connection header received
0929 16:29:26.654813 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.654819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.654944 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.655055 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.656816 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.657937 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.660135 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.660563 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.660681 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.664162 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.664167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.664171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.664204 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.666880 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.667654 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.667658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.667661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.667788 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.668179 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.668182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.668184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.668446 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.668563 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.669267 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.669496 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:29:26.671001 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.651931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44351 (local address 127.5.232.131:38743)
0929 16:29:26.652706 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:26.652752 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:26.654527 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:26.655304 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:26.655313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:26.655878 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:26.656642 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.656662 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.658121 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.658129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.659982 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.659999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.660763 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.660768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.660970 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.661714 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:26.661737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:26.663919 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:26.667044 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:26.667053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:26.667068 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:26.667531 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:26.667881 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:26.667884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:26.667887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:26.668088 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:26.668557 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:26.668563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:26.668918 (+ 355us) client_negotiation.cc:769] Sending connection context
0929 16:29:26.669183 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 16:29:26.669673 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":290,"threads_started":1}
I20260929 16:29:26.672714 12131 heartbeater.cc:347] Connected to a master server at 127.5.232.190:44351
I20260929 16:29:26.673041 12131 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:26.673818 12131 heartbeater.cc:511] Master 127.5.232.190:44351 requested a full tablet report, sending...
I20260929 16:29:26.675220 11674 ts_manager.cc:194] Registered new tserver with Master: a38da341d56b487f8d0138a6c3054e90 (127.5.232.131:33483)
I20260929 16:29:26.676119 11674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:38743
I20260929 16:29:26.685130 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/test-admin.keytab.
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.711309 6050 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 16:29:26.716564 12138 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:26.713867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.5.232.129:43393)
0929 16:29:26.714492 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.714503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.714603 (+ 100us) server_negotiation.cc:407] Connection header received
0929 16:29:26.714824 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.714832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.715058 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.715226 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.716120 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50958: BlockingRecv error: recv got EOF from 127.0.0.1:50958 (error 108)
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":269,"threads_started":1}
W20260929 16:29:26.716694 6050 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.718349 6050 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.719717 6050 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:29:26.734858 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.721473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.5.232.129:43393)
0929 16:29:26.721657 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.721664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.721831 (+ 167us) server_negotiation.cc:407] Connection header received
0929 16:29:26.721984 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.721988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.722045 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.722128 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.723061 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.724026 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.725347 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.725974 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.726072 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.729686 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.729702 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.729721 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.729763 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.732378 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.733048 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.733055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.733057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.733170 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.733521 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.733525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.733527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.733865 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.734086 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.734384 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.734590 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:29:26.741255 12138 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:26.739271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.5.232.129:43393)
0929 16:29:26.740011 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.740017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.740033 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:29:26.740085 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.740089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.740141 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.740240 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.741014 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50966: BlockingRecv error: recv got EOF from 127.0.0.1:50966 (error 108)
Metrics: {"server-negotiator.queue_time_us":567}
W20260929 16:29:26.741454 6050 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.742967 6050 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.744259 6050 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:29:26.758813 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.745616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.5.232.129:43393)
0929 16:29:26.745835 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.745841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.746097 (+ 256us) server_negotiation.cc:407] Connection header received
0929 16:29:26.746230 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.746235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.746294 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.746376 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.747242 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.748238 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.749463 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.749896 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.749991 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.754075 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.754081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.754084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.754137 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.756392 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.757074 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.757078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.757080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.757210 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.757604 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.757607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.757609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.757911 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.758102 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.758330 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.758485 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 16:29:26.763031 12138 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:26.761258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.5.232.129:43393)
0929 16:29:26.761486 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.761494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.761754 (+ 260us) server_negotiation.cc:407] Connection header received
0929 16:29:26.761926 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.761931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.762016 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.762111 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.762780 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50982: BlockingRecv error: recv got EOF from 127.0.0.1:50982 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 16:29:26.763247 6050 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699314866137-6050-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.764742 6050 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.766058 6050 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:29:26.779663 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.767311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.5.232.129:43393)
0929 16:29:26.767521 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.767527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.767713 (+ 186us) server_negotiation.cc:407] Connection header received
0929 16:29:26.767844 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.767848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.767905 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.768005 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.768950 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.769730 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.770885 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.771287 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.771375 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.774969 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.774975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.774978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.775010 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.777347 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.778003 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.778007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.778009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.778151 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.778443 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.778446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.778448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.778705 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.778915 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.779138 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.779300 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:29:26.781234 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 11736
I20260929 16:29:26.792861 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.781972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.5.232.129:43393)
0929 16:29:26.782134 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.782138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.782537 (+ 399us) server_negotiation.cc:407] Connection header received
0929 16:29:26.782663 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.782666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.782722 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.782798 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.783688 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.784347 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.785617 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.786004 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.786100 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.788175 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.788180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.788183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.788232 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.790776 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.791347 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.791351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.791354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.791463 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.791754 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.791757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.791759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.791984 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.792170 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.792450 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.792613 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:29:26.793569 11859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:26.876536 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 11736
I20260929 16:29:26.892835 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 11870
Sep 29 16:29:26 dist-test-slave-dssx krb5kdc[11630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:29:26.907928 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.893735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.5.232.130:39897)
0929 16:29:26.894305 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.894312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.894337 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:29:26.894399 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.894405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.894619 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.894757 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.895669 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.896651 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.897974 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.898520 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.898628 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.902379 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.902393 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.902409 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.902444 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.905281 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.905950 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.905957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.905960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.906080 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.906374 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.906377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.906379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.906823 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.907052 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.907326 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.907611 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":291,"threads_started":1}
I20260929 16:29:26.909214 11993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:26.992163 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 11870
I20260929 16:29:27.013015 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 12003
Sep 29 16:29:27 dist-test-slave-dssx krb5kdc[11630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:29:27.029714 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:27.014059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.5.232.131:33483)
0929 16:29:27.014677 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:27.014684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:27.014714 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:29:27.014773 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:27.014779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:27.014993 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:27.015180 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:27.016298 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:27.017309 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:27.018762 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:27.019361 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:27.019511 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:27.023686 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:27.023700 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:27.023716 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:27.023756 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:27.026681 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:27.027491 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:27.027500 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:27.027503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:27.027652 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:27.028058 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:27.028065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:27.028068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:27.028438 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:27.028783 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:27.029076 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:29:27.029355 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":306,"threads_started":1}
I20260929 16:29:27.031041 12126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:27.113674 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 12003
I20260929 16:29:27.129433 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 11643
I20260929 16:29:27.130616 11705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:27.232335 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 11643
2026-09-29T16:29:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3962 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:27 dist-test-slave-dssx krb5kdc[12158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:27 dist-test-slave-dssx krb5kdc[12158](info): set up 2 sockets
Sep 29 16:29:27 dist-test-slave-dssx krb5kdc[12158](info): commencing operation
krb5kdc: starting...
W20260929 16:29:28.326452 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:29:28 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.190.keytab.
Entry for principal oryx/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.190.keytab.
I20260929 16:29:28.534554 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:35773
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.190.keytab
--principal=oryx/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:35773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:28.862492 12171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:28.862879 12171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:28.863036 12171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:28.870441 12171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:28.870609 12171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:28.870744 12171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:28.870841 12171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:28.870911 12171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:28.880913 12171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:35773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.190.keytab
--principal=oryx/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:35773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:28.882401 12171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:28.883796 12171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:28.896204 12177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:28.896265 12179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:28.896286 12176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:28.897132 12171 server_base.cc:1034] running on GCE node
I20260929 16:29:28.898052 12171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:28.899598 12171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:28.900831 12171 hybrid_clock.cc:648] HybridClock initialized: now 1790699368900780 us; error 44 us; skew 500 ppm
I20260929 16:29:28.901209 12171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:28 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:28.904409 12171 init.cc:377] Logged in from keytab as oryx/127.5.232.190@KRBTEST.COM (short username oryx)
I20260929 16:29:28.905987 12171 webserver.cc:460] Webserver started at http://127.5.232.190:41745/ using document root <none> and password file <none>
I20260929 16:29:28.906680 12171 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:28.906783 12171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:28.907042 12171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:28.910161 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "c9daca0f91eb486fad1a5699e435f53a"
format_stamp: "Formatted at 2026-09-29 16:29:28 on dist-test-slave-dssx"
server_key: "f19e6be421a044a021148cae7c356967"
server_key_iv: "c7f49c1804636b5ef94c448a58e6b588"
server_key_version: "encryptionkey@0"
I20260929 16:29:28.910898 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "c9daca0f91eb486fad1a5699e435f53a"
format_stamp: "Formatted at 2026-09-29 16:29:28 on dist-test-slave-dssx"
server_key: "f19e6be421a044a021148cae7c356967"
server_key_iv: "c7f49c1804636b5ef94c448a58e6b588"
server_key_version: "encryptionkey@0"
I20260929 16:29:28.916504 12171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:29:28.920109 12186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:28.921666 12171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:29:28.921859 12171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "c9daca0f91eb486fad1a5699e435f53a"
format_stamp: "Formatted at 2026-09-29 16:29:28 on dist-test-slave-dssx"
server_key: "f19e6be421a044a021148cae7c356967"
server_key_iv: "c7f49c1804636b5ef94c448a58e6b588"
server_key_version: "encryptionkey@0"
I20260929 16:29:28.922058 12171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:28.940711 12171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:28.944880 12171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:28.945200 12171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:28.963986 12171 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:35773
I20260929 16:29:28.964010 12237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:35773 every 8 connection(s)
I20260929 16:29:28.965587 12171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:28.969643 12238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:28.974977 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 12171
I20260929 16:29:28.975225 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:28.975842 6050 external_mini_cluster.cc:1428] Setting key dbb441ce0b8a6e8a0b3ea684561f434d
I20260929 16:29:28.980870 12238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: Bootstrap starting.
I20260929 16:29:28.984858 12238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:28.986208 12238 log.cc:824] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:28.989912 12238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: No bootstrap required, opened a new log
Sep 29 16:29:28 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:28.997917 12238 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } }
I20260929 16:29:28.998340 12238 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:28.998466 12238 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9daca0f91eb486fad1a5699e435f53a, State: Initialized, Role: FOLLOWER
I20260929 16:29:28.999186 12238 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } }
I20260929 16:29:28.999449 12238 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:28.999585 12238 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:28.999790 12238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:29.000622 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:28.977899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.5.232.190:35773)
0929 16:29:28.978778 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:28.978791 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:28.978839 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:29:28.980510 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:28.980535 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:28.980948 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:28.981350 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:28.983251 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:28.984560 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:28.986414 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:28.987495 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:28.987691 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:28.992348 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:28.992356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:28.992372 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:28.992420 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:28.995739 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:28.996503 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:28.996510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:28.996517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:28.996708 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:28.997127 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:28.997132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:28.997134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:28.997788 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:28.998109 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:28.998645 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 16:29:28.999115 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":415,"threads_started":1}
I20260929 16:29:29.002840 12238 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } }
I20260929 16:29:29.003430 12238 leader_election.cc:302] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9daca0f91eb486fad1a5699e435f53a; no voters:
I20260929 16:29:29.004104 12238 leader_election.cc:288] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:29.004266 12243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:29.005007 12243 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 1 LEADER]: Becoming Leader. State: Replica: c9daca0f91eb486fad1a5699e435f53a, State: Running, Role: LEADER
I20260929 16:29:29.005585 12243 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } }
I20260929 16:29:29.006266 12238 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:29.008301 12244 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } }
I20260929 16:29:29.008749 12244 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:29.009622 12245 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9daca0f91eb486fad1a5699e435f53a. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } }
I20260929 16:29:29.010097 12245 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:29.013267 12248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:29.018812 12248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:29.027766 12248 catalog_manager.cc:1347] Generated new cluster ID: fa8e4544e6fb4a42ae8ab152fa0bb3fc
I20260929 16:29:29.027904 12248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:29.042568 12248 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:29.044489 12248 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:29.051975 12248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: Generated new TSK 0
I20260929 16:29:29.053002 12248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab.
Entry for principal oryx/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab.
I20260929 16:29:29.103240 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35773
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab
--principal=oryx/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:29.445359 12264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:29.445695 12264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:29.445850 12264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:29.453476 12264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:29.453598 12264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:29.453794 12264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:29.463929 12264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab
--principal=oryx/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:29.465395 12264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:29.466763 12264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:29.480341 12269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:29.480372 12272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:29.480420 12270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:29.480962 12264 server_base.cc:1034] running on GCE node
I20260929 16:29:29.481709 12264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:29.482712 12264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:29.483960 12264 hybrid_clock.cc:648] HybridClock initialized: now 1790699369483893 us; error 49 us; skew 500 ppm
I20260929 16:29:29.484398 12264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:29 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699369, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:29.487802 12264 init.cc:377] Logged in from keytab as oryx/127.5.232.129@KRBTEST.COM (short username oryx)
I20260929 16:29:29.489673 12264 webserver.cc:460] Webserver started at http://127.5.232.129:32805/ using document root <none> and password file <none>
I20260929 16:29:29.490370 12264 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:29.490473 12264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:29.490723 12264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:29.493705 12264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "9a026605c1054cf3a71540cc71a19432"
format_stamp: "Formatted at 2026-09-29 16:29:29 on dist-test-slave-dssx"
server_key: "772a3e7a30342f66fa78a00de972f74d"
server_key_iv: "4205c3e37767384a6106f5be10885725"
server_key_version: "encryptionkey@0"
I20260929 16:29:29.494434 12264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "9a026605c1054cf3a71540cc71a19432"
format_stamp: "Formatted at 2026-09-29 16:29:29 on dist-test-slave-dssx"
server_key: "772a3e7a30342f66fa78a00de972f74d"
server_key_iv: "4205c3e37767384a6106f5be10885725"
server_key_version: "encryptionkey@0"
I20260929 16:29:29.499740 12264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:29:29.503283 12279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:29.504696 12264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:29.504838 12264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "9a026605c1054cf3a71540cc71a19432"
format_stamp: "Formatted at 2026-09-29 16:29:29 on dist-test-slave-dssx"
server_key: "772a3e7a30342f66fa78a00de972f74d"
server_key_iv: "4205c3e37767384a6106f5be10885725"
server_key_version: "encryptionkey@0"
I20260929 16:29:29.505014 12264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:29.539908 12264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:29.544440 12264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:29.544694 12264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:29.545658 12264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:29.546979 12264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:29.547052 12264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:29.547155 12264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:29.547209 12264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:29.579290 12264 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:40865
I20260929 16:29:29.579335 12391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:40865 every 8 connection(s)
I20260929 16:29:29.580617 12264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:29.585424 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 12264
I20260929 16:29:29.585664 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:29.586457 6050 external_mini_cluster.cc:1428] Setting key 5d0014501a1e054cd0528a27c358dd67
Sep 29 16:29:29 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699369, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:29.602804 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:29.583171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:36865 (local address 127.5.232.190:35773)
0929 16:29:29.583585 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:29.583590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:29.584695 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 16:29:29.587328 (+ 2633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:29.587334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:29.587428 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:29.587558 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:29.589435 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:29.590160 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:29.592710 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:29.593085 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:29.593185 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:29.597155 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:29.597160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:29.597163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:29.597198 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:29.599741 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:29.600476 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:29.600483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:29.600486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:29.600601 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:29.601023 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:29.601028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:29.601031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:29.601295 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:29.601477 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:29.602190 (+ 713us) server_negotiation.cc:299] Negotiation successful
0929 16:29:29.602426 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":214,"threads_started":1}
I20260929 16:29:29.603884 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:29.583773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.129:36865)
0929 16:29:29.584507 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:29.584549 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:29.587020 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:29.587806 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:29.587815 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:29.588375 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:29.589197 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:29.589230 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:29.590326 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:29.590334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:29.592548 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:29.592566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:29.593332 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:29.593342 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:29.593721 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:29.594468 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:29.594520 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:29.596922 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:29.599891 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:29.599901 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:29.599919 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:29.600357 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:29.600714 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:29.600721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:29.600724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:29.600908 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:29.601454 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:29.601462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:29.601822 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 16:29:29.602100 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 16:29:29.602659 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":322,"threads_started":1}
I20260929 16:29:29.605623 12392 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35773
I20260929 16:29:29.605980 12392 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.5.232.130@KRBTEST.COM; defaulting to no policy
I20260929 16:29:29.606832 12392 heartbeater.cc:511] Master 127.5.232.190:35773 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.130@KRBTEST.COM" created.
I20260929 16:29:29.609171 12203 ts_manager.cc:194] Registered new tserver with Master: 9a026605c1054cf3a71540cc71a19432 (127.5.232.129:40865)
I20260929 16:29:29.610765 12203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:36865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab.
Entry for principal oryx/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab.
I20260929 16:29:29.632998 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35773
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab
--principal=oryx/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:29.970675 12398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:29.970988 12398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:29.971148 12398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:29.978832 12398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:29.978955 12398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:29.979141 12398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:29.989234 12398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab
--principal=oryx/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:29.990573 12398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:29.991870 12398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:30.005290 12406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:30.005290 12403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:30.005507 12404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:30.006088 12398 server_base.cc:1034] running on GCE node
I20260929 16:29:30.006853 12398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:30.007949 12398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:30.009143 12398 hybrid_clock.cc:648] HybridClock initialized: now 1790699370009089 us; error 49 us; skew 500 ppm
I20260929 16:29:30.009546 12398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:30 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699370, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:30.012728 12398 init.cc:377] Logged in from keytab as oryx/127.5.232.130@KRBTEST.COM (short username oryx)
I20260929 16:29:30.014353 12398 webserver.cc:460] Webserver started at http://127.5.232.130:35269/ using document root <none> and password file <none>
I20260929 16:29:30.015111 12398 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:30.015251 12398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:30.015547 12398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:30.018498 12398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "01ed68c31236410e9aac71d98835d8df"
format_stamp: "Formatted at 2026-09-29 16:29:30 on dist-test-slave-dssx"
server_key: "f79d09724bffe8fed07ec83e93b1b486"
server_key_iv: "a6a46b75f8f7e460dce8cda58dcb2117"
server_key_version: "encryptionkey@0"
I20260929 16:29:30.019243 12398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "01ed68c31236410e9aac71d98835d8df"
format_stamp: "Formatted at 2026-09-29 16:29:30 on dist-test-slave-dssx"
server_key: "f79d09724bffe8fed07ec83e93b1b486"
server_key_iv: "a6a46b75f8f7e460dce8cda58dcb2117"
server_key_version: "encryptionkey@0"
I20260929 16:29:30.024406 12398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:29:30.028127 12413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:30.029531 12398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:30.029706 12398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "01ed68c31236410e9aac71d98835d8df"
format_stamp: "Formatted at 2026-09-29 16:29:30 on dist-test-slave-dssx"
server_key: "f79d09724bffe8fed07ec83e93b1b486"
server_key_iv: "a6a46b75f8f7e460dce8cda58dcb2117"
server_key_version: "encryptionkey@0"
I20260929 16:29:30.029897 12398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:30.043308 12398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:30.047219 12398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:30.047528 12398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:30.048496 12398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:30.049929 12398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:30.050006 12398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:30.050165 12398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:30.050217 12398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:30.082641 12398 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:37611
I20260929 16:29:30.082693 12525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:37611 every 8 connection(s)
I20260929 16:29:30.083964 12398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:30.094278 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 12398
I20260929 16:29:30.094496 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:30.095034 6050 external_mini_cluster.cc:1428] Setting key ddb7235861d5c2d4fa54e214b99b9eac
Sep 29 16:29:30 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699370, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:30.106094 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:30.086671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:46533 (local address 127.5.232.190:35773)
0929 16:29:30.086884 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:30.086891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:30.088099 (+ 1208us) server_negotiation.cc:407] Connection header received
0929 16:29:30.089919 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:30.089924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:30.089995 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:30.090090 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:30.091625 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:30.092365 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:30.094099 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:30.094769 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:30.094910 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:30.099040 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:30.099045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:30.099049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:30.099082 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:30.102151 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:30.103212 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:30.103217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:30.103219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:30.103354 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:30.103901 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:30.103909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:30.103913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:30.104310 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:30.104465 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:30.105296 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 16:29:30.105588 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:29:30.106925 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:30.087219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.130:46533)
0929 16:29:30.087926 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:30.087965 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:30.089636 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:30.090321 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:30.090329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:30.090855 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:30.091455 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:30.091475 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:30.092530 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:30.092536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:30.093936 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:30.093949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:30.094978 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:30.094983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:30.095251 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:30.095929 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:30.095952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:30.098788 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:30.102342 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:30.102370 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:30.102393 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:30.103077 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:30.103467 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:30.103475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:30.103482 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:30.103767 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:30.104473 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:30.104482 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:30.104910 (+ 428us) client_negotiation.cc:769] Sending connection context
0929 16:29:30.105139 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 16:29:30.105620 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":272,"threads_started":1}
I20260929 16:29:30.108598 12526 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35773
I20260929 16:29:30.108948 12526 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:30.109826 12526 heartbeater.cc:511] Master 127.5.232.190:35773 requested a full tablet report, sending...
I20260929 16:29:30.111301 12203 ts_manager.cc:194] Registered new tserver with Master: 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611)
I20260929 16:29:30.112098 12203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:46533
WARNING: no policy specified for oryx/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab.
Entry for principal oryx/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab.
I20260929 16:29:30.144459 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35773
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab
--principal=oryx/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:30.468693 12531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:30.468997 12531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:30.469159 12531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:30.476881 12531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:30.477010 12531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:30.477218 12531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:30.487377 12531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab
--principal=oryx/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:30.488727 12531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:30.490080 12531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:30.503561 12536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:30.503577 12539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:30.503613 12537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:30.504801 12531 server_base.cc:1034] running on GCE node
I20260929 16:29:30.505539 12531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:30.506577 12531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:30.507788 12531 hybrid_clock.cc:648] HybridClock initialized: now 1790699370507723 us; error 43 us; skew 500 ppm
I20260929 16:29:30.508168 12531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:30 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699370, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:30.511364 12531 init.cc:377] Logged in from keytab as oryx/127.5.232.131@KRBTEST.COM (short username oryx)
I20260929 16:29:30.512856 12531 webserver.cc:460] Webserver started at http://127.5.232.131:40457/ using document root <none> and password file <none>
I20260929 16:29:30.513684 12531 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:30.513823 12531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:30.514112 12531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:30.517097 12531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "c28cf18f75454f029b1693e7ec5bf493"
format_stamp: "Formatted at 2026-09-29 16:29:30 on dist-test-slave-dssx"
server_key: "012ceb5948e86d9f82bf72de617354dd"
server_key_iv: "a53d128d485f54a48b2149b1893fae54"
server_key_version: "encryptionkey@0"
I20260929 16:29:30.517932 12531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "c28cf18f75454f029b1693e7ec5bf493"
format_stamp: "Formatted at 2026-09-29 16:29:30 on dist-test-slave-dssx"
server_key: "012ceb5948e86d9f82bf72de617354dd"
server_key_iv: "a53d128d485f54a48b2149b1893fae54"
server_key_version: "encryptionkey@0"
I20260929 16:29:30.523072 12531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:29:30.526590 12546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:30.528041 12531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:30.528214 12531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "c28cf18f75454f029b1693e7ec5bf493"
format_stamp: "Formatted at 2026-09-29 16:29:30 on dist-test-slave-dssx"
server_key: "012ceb5948e86d9f82bf72de617354dd"
server_key_iv: "a53d128d485f54a48b2149b1893fae54"
server_key_version: "encryptionkey@0"
I20260929 16:29:30.528421 12531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:30.571529 12531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:30.575525 12531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:30.575827 12531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:30.576761 12531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:30.578094 12531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:30.578197 12531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:30.578342 12531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:30.578414 12531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:30.610809 12531 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:39939
I20260929 16:29:30.610844 12658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:39939 every 8 connection(s)
I20260929 16:29:30.612069 12531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:30.614091 12392 heartbeater.cc:503] Master 127.5.232.190:35773 was elected leader, sending a full tablet report...
I20260929 16:29:30.614310 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 12531
I20260929 16:29:30.614570 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:30.615170 6050 external_mini_cluster.cc:1428] Setting key 2b06c17362c247b5a89558f44b597ef7
Sep 29 16:29:30 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699370, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:30.632432 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:30.614784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:47769 (local address 127.5.232.190:35773)
0929 16:29:30.615352 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:30.615360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:30.616896 (+ 1536us) server_negotiation.cc:407] Connection header received
0929 16:29:30.618883 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:30.618887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:30.618947 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:30.619035 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:30.620481 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:30.621234 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:30.622846 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:30.623240 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:30.623352 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:30.626701 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:30.626705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:30.626708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:30.626737 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:30.629569 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:30.630470 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:30.630475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:30.630477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:30.630624 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:30.631009 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:30.631012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:30.631014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:30.631254 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:30.631370 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:30.632007 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 16:29:30.632159 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":1032,"threads_started":1}
I20260929 16:29:30.633694 12662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:30.615448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.131:47769)
0929 16:29:30.616733 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:30.616779 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:30.618646 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:30.619259 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:30.619268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:30.619774 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:30.620346 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:30.620363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:30.621351 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:30.621355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:30.622618 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:30.622630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:30.623422 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:30.623426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:30.623615 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:30.624309 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:30.624331 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:30.626523 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:30.629694 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:30.629701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:30.629714 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:30.630325 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:30.630730 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:30.630734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:30.630736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:30.630916 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:30.631366 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:30.631372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:30.631690 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 16:29:30.631908 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 16:29:30.632399 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":834,"threads_started":1}
I20260929 16:29:30.635036 12659 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35773
I20260929 16:29:30.635325 12659 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:30.636008 12659 heartbeater.cc:511] Master 127.5.232.190:35773 requested a full tablet report, sending...
I20260929 16:29:30.637337 12203 ts_manager.cc:194] Registered new tserver with Master: c28cf18f75454f029b1693e7ec5bf493 (127.5.232.131:39939)
I20260929 16:29:30.638142 12203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:47769
I20260929 16:29:30.642503 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.232.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.189.keytab.
Entry for principal oryx/127.5.232.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.189.keytab.
I20260929 16:29:30.689226 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.189:42019
--webserver_interface=127.5.232.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.189.keytab
--principal=oryx/127.5.232.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:35773,127.5.232.189:42019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:31.022593 12665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:31.022879 12665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:31.022964 12665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:31.030499 12665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:31.030628 12665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:31.030673 12665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:31.030728 12665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:31.030771 12665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:31.040984 12665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:35773,127.5.232.189:42019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.189.keytab
--principal=oryx/127.5.232.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.189:42019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.232.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
W20260929 16:29:31.041316 12665 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.232.190:35773,127.5.232.189:42019'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 16:29:31.042531 12665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:31.043730 12665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:31.055188 12670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:31.055104 12671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:31.055159 12673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:31.056264 12665 server_base.cc:1034] running on GCE node
I20260929 16:29:31.057022 12665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:31.058463 12665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:31.059662 12665 hybrid_clock.cc:648] HybridClock initialized: now 1790699371059583 us; error 57 us; skew 500 ppm
I20260929 16:29:31.059969 12665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699371, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:31.062924 12665 init.cc:377] Logged in from keytab as oryx/127.5.232.189@KRBTEST.COM (short username oryx)
I20260929 16:29:31.064373 12665 webserver.cc:460] Webserver started at http://127.5.232.189:43183/ using document root <none> and password file <none>
I20260929 16:29:31.065042 12665 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:31.065126 12665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:31.065519 12665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:31.068403 12665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data/instance:
uuid: "6721b56a80e64285978fa1b321a9f761"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "4330cf544cdf37c7682b38adda238c22"
server_key_iv: "ed3d70ae9b77f1c6d98cd8fd1fb7a9ec"
server_key_version: "encryptionkey@0"
I20260929 16:29:31.069154 12665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal/instance:
uuid: "6721b56a80e64285978fa1b321a9f761"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "4330cf544cdf37c7682b38adda238c22"
server_key_iv: "ed3d70ae9b77f1c6d98cd8fd1fb7a9ec"
server_key_version: "encryptionkey@0"
I20260929 16:29:31.074177 12665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:29:31.077610 12680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:31.078960 12665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:29:31.079131 12665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal
uuid: "6721b56a80e64285978fa1b321a9f761"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "4330cf544cdf37c7682b38adda238c22"
server_key_iv: "ed3d70ae9b77f1c6d98cd8fd1fb7a9ec"
server_key_version: "encryptionkey@0"
I20260929 16:29:31.079331 12665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:31.103864 12665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:31.107864 12665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:31.108150 12665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:31.115049 12526 heartbeater.cc:503] Master 127.5.232.190:35773 was elected leader, sending a full tablet report...
I20260929 16:29:31.126310 12665 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.189:42019
I20260929 16:29:31.126353 12731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.189:42019 every 8 connection(s)
I20260929 16:29:31.127786 12665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
I20260929 16:29:31.129176 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 12665
I20260929 16:29:31.132534 12732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:31.141448 12732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.189@KRBTEST.COM
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699371, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.189@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:31.159360 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.131401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.5.232.189:42019)
0929 16:29:31.132165 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.132181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.132239 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:29:31.134620 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.134649 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.135139 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.135614 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.136844 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.138922 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.140106 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.141131 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.141321 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.145970 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.145981 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.145994 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.146033 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.154589 (+ 8556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.155353 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.155364 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.155373 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.155560 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.156012 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.156018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.156022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.156796 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.157105 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.157520 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.157985 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":305,"threads_started":1}
I20260929 16:29:31.160902 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.143964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57316)
0929 16:29:31.144812 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.144845 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.145022 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.145421 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.145428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.146201 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.146553 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.146565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.147764 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.147768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.148794 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.148806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.149486 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.149491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.149617 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.152053 (+ 2436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.152081 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.154772 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.158423 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.158432 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.158436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.158860 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.159219 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.159222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.159224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.159360 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.159988 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.159994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.160130 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.160284 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.160510 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":496,"threads_started":1}
I20260929 16:29:31.160929 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.143968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.5.232.190:35773)
0929 16:29:31.144491 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.144500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.144941 (+ 441us) server_negotiation.cc:407] Connection header received
0929 16:29:31.145142 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.145147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.145215 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.145338 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.146744 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.147636 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.148913 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.149284 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.149437 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.154973 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.154978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.154981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.155018 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.158293 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.158977 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.158985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.158990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.159120 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.159459 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.159464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.159467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.159868 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.160025 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.160392 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.160611 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":272,"threads_started":1}
I20260929 16:29:31.162555 12732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699371, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.189@KRBTEST.COM for oryx/127.5.232.189@KRBTEST.COM
I20260929 16:29:31.173748 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.163364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.0.0.1:49066)
0929 16:29:31.163519 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.163538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.163632 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.163881 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.163883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.164189 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.164317 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.164324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.165212 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.165216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.166121 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.166131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.166753 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.166757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.166902 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.167461 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.167477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.169352 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.171835 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.171839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.171841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.172192 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.172492 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.172495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.172497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.172606 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.173021 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.173024 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.173139 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.173245 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.173504 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 16:29:31.173990 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.163276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.5.232.189:42019)
0929 16:29:31.163427 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.163431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.163596 (+ 165us) server_negotiation.cc:407] Connection header received
0929 16:29:31.163711 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.163715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.163775 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.163849 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.164422 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.165098 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.166245 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.166638 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.166737 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.169537 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.169541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.169544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.169569 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.171713 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.172282 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.172286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.172288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.172406 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.172686 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.172689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.172691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.172937 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.173034 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.173315 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.173635 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":52}
I20260929 16:29:31.178548 12732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Bootstrap starting.
I20260929 16:29:31.181746 12732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:31.182814 12732 log.cc:824] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:31.185534 12732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: No bootstrap required, opened a new log
I20260929 16:29:31.192539 12732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } }
I20260929 16:29:31.192945 12732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:31.193054 12732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6721b56a80e64285978fa1b321a9f761, State: Initialized, Role: FOLLOWER
I20260929 16:29:31.193704 12732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } }
I20260929 16:29:31.194528 12739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } } }
I20260929 16:29:31.194809 12739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:29:31.195595 12732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:29:31.204458 12750 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:29:31.204651 12750 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:29:31.206328 12665 master_runner.cc:428] Detected that this master 6721b56a80e64285978fa1b321a9f761 is joining an existing cluster
I20260929 16:29:31.206440 12665 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.232.189:42019
I20260929 16:29:31.227377 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.210566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57328)
0929 16:29:31.211233 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.211251 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.211378 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.211809 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.211812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.212317 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.212628 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.212636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.214001 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.214009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.216050 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.216068 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.217516 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.217525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.217764 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.218486 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.218515 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.219886 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.223872 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.223876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.223879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.224349 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.224751 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.224759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.224764 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.224956 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.225805 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.225812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.226078 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.226268 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.226878 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"spinlock_wait_cycles":32640,"thread_start_us":637,"threads_started":1}
I20260929 16:29:31.227730 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.211128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.5.232.190:35773)
0929 16:29:31.211372 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.211381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.211407 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:29:31.211520 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.211524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.211591 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.211647 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.212781 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.213841 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.216224 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.217051 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.217171 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.220104 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.220114 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.220122 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.220168 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.223734 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.224464 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.224469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.224473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.224625 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.225101 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.225108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.225113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.225553 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.225701 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.226793 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.227334 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:29:31.233634 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.211023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.5.232.189:42019)
0929 16:29:31.211198 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.211204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.217766 (+ 6562us) server_negotiation.cc:407] Connection header received
0929 16:29:31.217990 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.217996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.218065 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.218132 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.219670 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.220736 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.222697 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.223473 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.223615 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.226168 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.226175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.226180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.226211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.230377 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.231171 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.231181 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.231184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.231334 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.231778 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.231783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.231786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.232197 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.232366 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.232921 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.233242 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":64}
I20260929 16:29:31.233675 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.210729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.0.0.1:49070)
0929 16:29:31.217677 (+ 6948us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.217697 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.217862 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.218319 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.218326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.219142 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.219495 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.219506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.220951 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.220958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.222541 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.222555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.223678 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.223685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.223879 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.224863 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.224886 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.225976 (+ 1090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.230519 (+ 4543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.230524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.230528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.231051 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.231511 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.231516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.231519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.231665 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.232429 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.232434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.232630 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.232776 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.233077 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6562,"thread_start_us":6653,"threads_started":1}
WARNING: no policy specified for oryx/127.5.232.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.188@KRBTEST.COM" created.
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.190@KRBTEST.COM for oryx/127.5.232.189@KRBTEST.COM
I20260929 16:29:31.256034 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.234446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.0.0.1:49072)
0929 16:29:31.237935 (+ 3489us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.237984 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.238202 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.238729 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.238738 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.239611 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.239953 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.239966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.241637 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.241642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.243227 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.243242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.244233 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.244240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.244446 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.245190 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.245214 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.248300 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.252605 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.252623 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.252635 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.253164 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.253622 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.253627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.253630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.253774 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.254750 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.254759 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.255013 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.255232 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.255578 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3227,"thread_start_us":3086,"threads_started":1}
I20260929 16:29:31.256071 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.235303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.5.232.189:42019)
0929 16:29:31.235854 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.235860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.238066 (+ 2206us) server_negotiation.cc:407] Connection header received
0929 16:29:31.238420 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.238428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.238543 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.238653 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.240128 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.241503 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.243380 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.243898 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.244057 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.248521 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.248528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.248533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.248571 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.252435 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.253306 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.253314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.253319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.253501 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.253883 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.253889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.253893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.254191 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.254737 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.255346 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.255618 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434}
I20260929 16:29:31.258800 12203 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.232.189:42019: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6721b56a80e64285978fa1b321a9f761"
member_type: NON_VOTER
last_known_addr {
host: "127.5.232.189"
port: 42019
}
attrs {
promote: true
}
}
dest_uuid: "c9daca0f91eb486fad1a5699e435f53a"
cas_config_opid_index: -1
I20260929 16:29:31.260017 12203 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } }
I20260929 16:29:31.269506 12762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6721b56a80e64285978fa1b321a9f761 (127.5.232.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } } }
I20260929 16:29:31.271175 12707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:31.273240 12762 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.232.189:42019
I20260929 16:29:31.274080 12665 master.cc:537] Master@127.5.232.189:42019 shutting down...
I20260929 16:29:31.274883 12763 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } } }
I20260929 16:29:31.275318 12763 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:31.275483 12707 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Refusing update from remote peer c9daca0f91eb486fad1a5699e435f53a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 16:29:31.278476 12763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.188.keytab.
Entry for principal oryx/127.5.232.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.188.keytab.
I20260929 16:29:31.281189 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.188:34743
--webserver_interface=127.5.232.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.188.keytab
--principal=oryx/127.5.232.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:35773,127.5.232.189:42019,127.5.232.188:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:31.281886 12188 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 16:29:31.287894 12665 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:31.288602 12665 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:31.288851 12665 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: stopping tablet replica
I20260929 16:29:31.295660 12665 master.cc:552] Master@127.5.232.189:42019 shutdown complete.
I20260929 16:29:31.295805 12665 master_runner.cc:305] Clearing existing system tablet
I20260929 16:29:31.296295 12665 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:31.296454 12665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:31.299521 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.299050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.0.0.1:49088)
0929 16:29:31.299232 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.299349 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:29:31.299777 12188 consensus_peers.cc:487] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a -> Peer 6721b56a80e64285978fa1b321a9f761 (127.5.232.189:42019): Couldn't send request to peer 6721b56a80e64285978fa1b321a9f761. Status: Network error: Client connection negotiation failed: client connection to 127.5.232.189:42019: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:29:31.302089 12665 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260929 16:29:31.306164 12768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:31.306528 12665 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:31.306798 12665 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 16:29:31.306996 12665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal
uuid: "6721b56a80e64285978fa1b321a9f761"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "4330cf544cdf37c7682b38adda238c22"
server_key_iv: "ed3d70ae9b77f1c6d98cd8fd1fb7a9ec"
server_key_version: "encryptionkey@0"
I20260929 16:29:31.312211 12665 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:31.319283 12665 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:29:31.319464 12665 log.cc:1196] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 16:29:31.319917 12665 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Deleting consensus metadata
I20260929 16:29:31.320173 12665 master_runner.cc:315] Copying system tablet from 127.5.232.190:35773
I20260929 16:29:31.322649 12665 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: tablet copy: Beginning tablet copy session from remote peer at address 127.5.232.190:35773
I20260929 16:29:31.344329 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.323524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.5.232.190:35773)
0929 16:29:31.323713 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.323721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.329991 (+ 6270us) server_negotiation.cc:407] Connection header received
0929 16:29:31.330198 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.330204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.330298 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.330417 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.331637 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.332800 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.334660 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.335339 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.335470 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.337802 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.337810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.337815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.337862 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.341260 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.342103 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.342109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.342113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.342260 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.342684 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.342689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.342693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.343049 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.343192 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.343705 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.343939 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:29:31.344355 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.323471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57330)
0929 16:29:31.329828 (+ 6357us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.329860 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.330044 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.330635 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.330640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.331210 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.331468 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.331481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.332949 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.332955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.334524 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.334540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.335526 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.335532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.335724 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.336430 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.336453 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.337602 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.341440 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.341445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.341449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.341946 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.342378 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.342384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.342387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.342557 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.343177 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.343182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.343413 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.343561 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.343868 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6207,"thread_start_us":6101,"threads_started":1}
I20260929 16:29:31.345631 12223 tablet_copy_service.cc:140] P c9daca0f91eb486fad1a5699e435f53a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6721b56a80e64285978fa1b321a9f761 ({username='oryx', principal='oryx/127.5.232.189@KRBTEST.COM'} at 127.0.0.1:57330)
I20260929 16:29:31.345909 12223 tablet_copy_service.cc:161] P c9daca0f91eb486fad1a5699e435f53a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6721b56a80e64285978fa1b321a9f761 at {username='oryx', principal='oryx/127.5.232.189@KRBTEST.COM'} at 127.0.0.1:57330: session id = 6721b56a80e64285978fa1b321a9f761-00000000000000000000000000000000
I20260929 16:29:31.350479 12223 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:29:31.352800 12665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:29:31.359648 12665 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: tablet copy: Starting download of 0 data blocks...
I20260929 16:29:31.359964 12665 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: tablet copy: Starting download of 1 WAL segments...
I20260929 16:29:31.362768 12665 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:29:31.367228 12223 tablet_copy_service.cc:342] P c9daca0f91eb486fad1a5699e435f53a: Request end of tablet copy session 6721b56a80e64285978fa1b321a9f761-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.232.189@KRBTEST.COM'} at 127.0.0.1:57330
I20260929 16:29:31.367440 12223 tablet_copy_service.cc:433] P c9daca0f91eb486fad1a5699e435f53a: ending tablet copy session 6721b56a80e64285978fa1b321a9f761-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6721b56a80e64285978fa1b321a9f761
I20260929 16:29:31.379516 12665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:31.383201 12776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:31.384052 12775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:31.385347 12778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:31.391115 12665 server_base.cc:1034] running on GCE node
I20260929 16:29:31.391500 12665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:31.392297 12665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:31.393553 12665 hybrid_clock.cc:648] HybridClock initialized: now 1790699371393476 us; error 39 us; skew 500 ppm
I20260929 16:29:31.393797 12665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699371, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:31.396109 12665 init.cc:377] Logged in from keytab as oryx/127.5.232.189@KRBTEST.COM (short username oryx)
I20260929 16:29:31.400794 12665 webserver.cc:460] Webserver started at http://127.5.232.189:40731/ using document root <none> and password file <none>
I20260929 16:29:31.401172 12665 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:31.401265 12665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:31.404474 12665 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:29:31.412756 12785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:31.413525 12665 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.004s
I20260929 16:29:31.413735 12665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal
uuid: "6721b56a80e64285978fa1b321a9f761"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "4330cf544cdf37c7682b38adda238c22"
server_key_iv: "ed3d70ae9b77f1c6d98cd8fd1fb7a9ec"
server_key_version: "encryptionkey@0"
I20260929 16:29:31.413950 12665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:31.430706 12665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:31.434903 12665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:31.464848 12665 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.189:42019
I20260929 16:29:31.464912 12836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.189:42019 every 8 connection(s)
I20260929 16:29:31.465896 12665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
I20260929 16:29:31.471539 12837 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:29:31.475477 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Bootstrap starting.
I20260929 16:29:31.491397 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:29:31.491904 12837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Bootstrap complete.
I20260929 16:29:31.492923 12837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } }
I20260929 16:29:31.493171 12837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6721b56a80e64285978fa1b321a9f761, State: Initialized, Role: LEARNER
I20260929 16:29:31.493475 12837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } }
I20260929 16:29:31.494369 12840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } } }
I20260929 16:29:31.494612 12837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:31.494750 12840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:29:31.502835 12851 catalog_manager.cc:1259] Loaded cluster ID: fa8e4544e6fb4a42ae8ab152fa0bb3fc
I20260929 16:29:31.502964 12851 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: loading cluster ID for follower catalog manager: success
I20260929 16:29:31.507697 12851 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: acquiring CA information for follower catalog manager: success
I20260929 16:29:31.510720 12851 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:29:31.640862 12659 heartbeater.cc:503] Master 127.5.232.190:35773 was elected leader, sending a full tablet report...
I20260929 16:29:31.686829 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.679363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.0.0.1:49100)
0929 16:29:31.679554 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.679576 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.679712 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.680557 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.680562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.680592 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:31.680783 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.680793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.682608 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.682614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.684622 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.684638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.685926 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.685934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.686167 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.686231 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.686356 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.686512 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 16:29:31.686914 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.679627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.5.232.189:42019)
0929 16:29:31.680217 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.680221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.680242 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:29:31.680302 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.680306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.680375 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.680478 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:31.680925 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.682439 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.684793 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.685734 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.685841 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.686018 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.686411 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.686544 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":354,"threads_started":1}
I20260929 16:29:31.688493 12812 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 16:29:31.689044 12840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9daca0f91eb486fad1a5699e435f53a. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } } }
I20260929 16:29:31.689357 12840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: This master's current role is: LEARNER
W20260929 16:29:31.696331 12766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:31.696659 12766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:31.696841 12766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:31.704594 12766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:31.704698 12766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:31.704799 12766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:31.704897 12766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:31.704979 12766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:31.715723 12766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:35773,127.5.232.189:42019,127.5.232.188:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.188.keytab
--principal=oryx/127.5.232.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.188:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.232.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:31.717218 12766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:31.718535 12766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:31.734000 12858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:31.734045 12857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:31.734242 12860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:31.734876 12766 server_base.cc:1034] running on GCE node
I20260929 16:29:31.735723 12766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:31.737226 12766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:31.738457 12766 hybrid_clock.cc:648] HybridClock initialized: now 1790699371738407 us; error 66 us; skew 500 ppm
I20260929 16:29:31.738813 12766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699371, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:31.741947 12766 init.cc:377] Logged in from keytab as oryx/127.5.232.188@KRBTEST.COM (short username oryx)
I20260929 16:29:31.743399 12766 webserver.cc:460] Webserver started at http://127.5.232.188:38541/ using document root <none> and password file <none>
I20260929 16:29:31.744091 12766 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:31.744257 12766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:31.744554 12766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:31.747562 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data/instance:
uuid: "0b43f0470b7c49c593a6dc06d8c12162"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "2854a5bba41ac2c4453654f22b2aa780"
server_key_iv: "93e9b00a7b651de5215479eddd8b2767"
server_key_version: "encryptionkey@0"
I20260929 16:29:31.748330 12766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal/instance:
uuid: "0b43f0470b7c49c593a6dc06d8c12162"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "2854a5bba41ac2c4453654f22b2aa780"
server_key_iv: "93e9b00a7b651de5215479eddd8b2767"
server_key_version: "encryptionkey@0"
I20260929 16:29:31.753496 12766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:29:31.756790 12867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:31.758087 12766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:29:31.758270 12766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal
uuid: "0b43f0470b7c49c593a6dc06d8c12162"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "2854a5bba41ac2c4453654f22b2aa780"
server_key_iv: "93e9b00a7b651de5215479eddd8b2767"
server_key_version: "encryptionkey@0"
I20260929 16:29:31.758471 12766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:31.788529 12766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:31.793000 12766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:31.793303 12766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:31.813748 12766 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.188:34743
I20260929 16:29:31.813767 12918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.188:34743 every 8 connection(s)
I20260929 16:29:31.815315 12766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
I20260929 16:29:31.818132 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 12766
I20260929 16:29:31.819657 12919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:31.830924 12919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.188@KRBTEST.COM
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699371, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.188@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:31.849475 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.820205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.5.232.188:34743)
0929 16:29:31.821007 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.821019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.825946 (+ 4927us) server_negotiation.cc:407] Connection header received
0929 16:29:31.827852 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.827879 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.828399 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.828967 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.831047 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.832652 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.834600 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.835617 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.835819 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.840702 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.840716 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.840733 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.840783 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.844925 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.845829 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.845839 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.845850 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.846051 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.846417 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.846423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.846427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.847152 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.847485 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.847931 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.848360 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":597,"thread_start_us":367,"threads_started":1}
I20260929 16:29:31.849836 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.833712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57346)
0929 16:29:31.834523 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.834572 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.834789 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.835157 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.835162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.835810 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.836081 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.836090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.837139 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.837147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.838247 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.838260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.839046 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.839054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.839208 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.839937 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.839973 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.843393 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.846995 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.847004 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.847010 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.847450 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.847862 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.847866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.847870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.848004 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.848601 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.848608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.848788 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.848985 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.849308 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":417,"threads_started":1}
I20260929 16:29:31.850817 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.833762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.5.232.190:35773)
0929 16:29:31.833962 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.833968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.834679 (+ 711us) server_negotiation.cc:407] Connection header received
0929 16:29:31.834906 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.834912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.834993 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.835097 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.836245 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.837003 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.838420 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.838817 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.838956 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.843610 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.843620 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.843626 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.843672 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.846875 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.847604 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.847609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.847612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.847745 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.848148 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.848153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.848156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.848468 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.848627 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.849089 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.849315 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:29:31.851972 12919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699371, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.188@KRBTEST.COM for oryx/127.5.232.189@KRBTEST.COM
I20260929 16:29:31.867697 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.852775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.0.0.1:49102)
0929 16:29:31.852987 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.853002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.853092 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.853350 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.853353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.853718 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.853846 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.853853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.854737 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.854740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.855631 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.855641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.856291 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.856295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.856414 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.860675 (+ 4261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.860703 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.862842 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.865739 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.865743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.865746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.866121 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.866449 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.866453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.866455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.866575 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.867017 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.867020 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.867135 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.867243 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.867458 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:29:31.867723 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.852851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.5.232.189:42019)
0929 16:29:31.853049 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.853065 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.853082 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:29:31.853183 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.853187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.853242 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.853329 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.853964 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.854627 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.855770 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.856164 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.856256 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.863054 (+ 6798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.863060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.863063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.863095 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.865610 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.866225 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.866229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.866231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.866358 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.866662 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.866665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.866667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.866923 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.867030 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.867334 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.867504 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:29:31.869197 12919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699371, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.188@KRBTEST.COM for oryx/127.5.232.188@KRBTEST.COM
I20260929 16:29:31.880951 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.869938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:34743 (local address 127.0.0.1:35852)
0929 16:29:31.870119 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.870143 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.870238 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.870496 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.870499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.870832 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.870971 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.870978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.871866 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.871870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.872721 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.872731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.873339 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.873344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.873581 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.874184 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.874201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.876146 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.878909 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.878913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.878916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.879255 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.879593 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.879596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.879599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.879736 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.880165 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.880169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.880313 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.880420 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.880611 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:29:31.880992 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.869989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.5.232.188:34743)
0929 16:29:31.870140 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.870144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.870185 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:29:31.870332 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.870335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.870387 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.870462 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.871083 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.871752 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.872830 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.873209 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.873310 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.876330 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.876335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.876338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.876365 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.878776 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.879371 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.879375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.879377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.879501 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.879826 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.879829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.879831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.880073 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.880180 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.880507 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.880743 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":56}
I20260929 16:29:31.885869 12919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: Bootstrap starting.
I20260929 16:29:31.889030 12919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:31.890595 12919 log.cc:824] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:31.893225 12919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: No bootstrap required, opened a new log
I20260929 16:29:31.900679 12919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } }
I20260929 16:29:31.901057 12919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:31.901199 12919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b43f0470b7c49c593a6dc06d8c12162, State: Initialized, Role: FOLLOWER
I20260929 16:29:31.901901 12919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } }
I20260929 16:29:31.902778 12925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } } }
I20260929 16:29:31.903218 12925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:29:31.904084 12919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:29:31.914286 12936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:29:31.914494 12936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:29:31.919036 12766 master_runner.cc:428] Detected that this master 0b43f0470b7c49c593a6dc06d8c12162 is joining an existing cluster
I20260929 16:29:31.919144 12766 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.232.188:34743
I20260929 16:29:31.944399 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.923044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.5.232.190:35773)
0929 16:29:31.923250 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.923255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.924459 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 16:29:31.924758 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.924764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.924853 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.924936 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.929071 (+ 4135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.930249 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.932616 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.933273 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.933417 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.937563 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.937570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.937575 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.937618 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.941265 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.942114 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.942121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.942127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.942289 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.942773 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.942778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.942781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.943119 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.943231 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.943819 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.944042 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:29:31.944440 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.923545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.0.0.1:49104)
0929 16:29:31.924660 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.924678 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.924869 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.925426 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.925432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.927421 (+ 1989us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.927776 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.927790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.929702 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.929724 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.932525 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.932550 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.934334 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.934340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.934584 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.935419 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.935452 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.937099 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.941126 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.941132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.941136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.941763 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.942239 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.942244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.942247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.942429 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.943166 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.943173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.943409 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.943580 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.943902 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":49,"thread_start_us":814,"threads_started":1}
I20260929 16:29:31.945847 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.923150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.5.232.189:42019)
0929 16:29:31.923360 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.923368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.924762 (+ 1394us) server_negotiation.cc:407] Connection header received
0929 16:29:31.925024 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.925031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.925107 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.925178 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.927983 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.929445 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.932770 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.933744 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.933881 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.937409 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.937417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.937423 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.937465 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.940960 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.941919 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.941926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.941930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.942093 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.942575 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.942580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.942583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.942958 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.943089 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.944840 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.945080 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:29:31.946557 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.923477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57354)
0929 16:29:31.924323 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.924356 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.924580 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.925637 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.925641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.928530 (+ 2889us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.928888 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.928900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.930444 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.930449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.932439 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.932454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.933751 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.933758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.933984 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.934776 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.934804 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.937313 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.941448 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.941453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.941456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.941970 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.942432 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.942436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.942439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.942618 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.943319 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.943324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.943499 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.945144 (+ 1645us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.946080 (+ 936us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":49,"thread_start_us":444,"threads_started":1}
I20260929 16:29:31.954912 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.924755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:34743 (local address 127.0.0.1:35864)
0929 16:29:31.936956 (+ 12201us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.936976 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.937115 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.937603 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.937607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.938294 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.938697 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.938712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.941190 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.941196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.942857 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.942888 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.944194 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.944202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.944392 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.945033 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.945057 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.946667 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.950854 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.950862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.950867 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.951393 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.951781 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.951789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.951796 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.951948 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.953691 (+ 1743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.953698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.953870 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.954022 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.954381 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12041,"thread_start_us":11444,"threads_started":1}
I20260929 16:29:31.954942 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.925054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35864 (local address 127.5.232.188:34743)
0929 16:29:31.925669 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.925674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.937085 (+ 11411us) server_negotiation.cc:407] Connection header received
0929 16:29:31.937257 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.937262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.937332 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.937455 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.938866 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.940048 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.943031 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.943778 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.943873 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.946864 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.946871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.946875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.946915 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.950714 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.951539 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.951545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.951548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.951677 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.952702 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.952708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.952711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.953583 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.953736 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.954167 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.954566 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":486}
I20260929 16:29:31.955724 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 12264
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.190@KRBTEST.COM for oryx/127.5.232.188@KRBTEST.COM
Sep 29 16:29:31 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.129@KRBTEST.COM
I20260929 16:29:31.969439 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.951938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:34743 (local address 127.0.0.1:35878)
0929 16:29:31.952348 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:31.952368 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:31.952491 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:31.953300 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:31.953308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:31.953796 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:31.953975 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.953989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.955644 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.955650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.957510 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:31.957525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.958574 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.958582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:31.958775 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.959428 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:31.959445 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:31.961990 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:31.966501 (+ 4511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.966506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.966511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.967026 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.967519 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:31.967525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:31.967529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:31.967719 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:31.968390 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:31.968395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:31.968581 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 16:29:31.968716 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:29:31.969017 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260929 16:29:31.970481 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.952018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.5.232.188:34743)
0929 16:29:31.952583 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.952587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.952603 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:29:31.952669 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.952672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.952734 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.953281 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.954160 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.955463 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.957739 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.958396 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.959113 (+ 717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.962234 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.962242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.962247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.962284 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.966337 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.967192 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.967198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.967202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.967380 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.967864 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.967875 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.967878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.968232 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.969684 (+ 1452us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.969840 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.970058 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":555,"threads_started":1}
I20260929 16:29:31.972136 12203 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.232.188:34743: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162"
member_type: NON_VOTER
last_known_addr {
host: "127.5.232.188"
port: 34743
}
attrs {
promote: true
}
}
dest_uuid: "c9daca0f91eb486fad1a5699e435f53a"
cas_config_opid_index: 5
I20260929 16:29:31.973097 12203 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } }
I20260929 16:29:31.975040 12763 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0b43f0470b7c49c593a6dc06d8c12162 (127.5.232.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:31.979956 12763 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.232.188:34743
I20260929 16:29:31.980201 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:31.957369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.5.232.129:40865)
0929 16:29:31.958071 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:31.958080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:31.958111 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:29:31.958191 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:31.958201 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:31.958505 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:31.958675 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:31.960370 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.961629 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.963603 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:31.964535 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:31.970966 (+ 6431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:31.971252 (+ 286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:31.971264 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:31.971283 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:31.971316 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:31.975852 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.976871 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.976881 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.976884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.977053 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:31.977542 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:31.977547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:31.977551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:31.977986 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:31.978295 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:31.979513 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0929 16:29:31.979793 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":340,"threads_started":1}
I20260929 16:29:31.980726 12766 master.cc:537] Master@127.5.232.188:34743 shutting down...
I20260929 16:29:31.981766 12387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:31.982664 12812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 LEARNER]: Refusing update from remote peer c9daca0f91eb486fad1a5699e435f53a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:29:32.003410 12947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.004856 12948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 16:29:32.005182 12947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
W20260929 16:29:32.005916 12188 consensus_peers.cc:487] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a -> Peer 0b43f0470b7c49c593a6dc06d8c12162 (127.5.232.188:34743): Couldn't send request to peer 0b43f0470b7c49c593a6dc06d8c12162. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:29:32.009402 12812 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0b43f0470b7c49c593a6dc06d8c12162 (127.5.232.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.012712 12840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: NON_VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: true } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.013026 12840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:29:32.016153 12947 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: attempting to promote NON_VOTER 6721b56a80e64285978fa1b321a9f761 to VOTER
I20260929 16:29:32.017117 12947 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } }
I20260929 16:29:32.019250 12766 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:32.019726 12766 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:32.020006 12766 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: stopping tablet replica
I20260929 16:29:32.021135 12812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 LEARNER]: Refusing update from remote peer c9daca0f91eb486fad1a5699e435f53a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260929 16:29:32.021659 12188 consensus_peers.cc:487] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a -> Peer 0b43f0470b7c49c593a6dc06d8c12162 (127.5.232.188:34743): Couldn't send request to peer 0b43f0470b7c49c593a6dc06d8c12162. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:29:32.022001 12948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 16:29:32.024931 12947 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6721b56a80e64285978fa1b321a9f761 (127.5.232.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.027715 12766 master.cc:552] Master@127.5.232.188:34743 shutdown complete.
I20260929 16:29:32.027864 12766 master_runner.cc:305] Clearing existing system tablet
I20260929 16:29:32.028285 12766 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:32.028424 12766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:32.030133 12812 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6721b56a80e64285978fa1b321a9f761 (127.5.232.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.034526 12840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.034821 12840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:29:32.036144 12763 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.036440 12763 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:32.036888 12947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.037130 12947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:32.038004 12766 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.004s
I20260929 16:29:32.040160 12954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:32.040495 12766 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:32.040933 12766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:29:32.041092 12766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal
uuid: "0b43f0470b7c49c593a6dc06d8c12162"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "2854a5bba41ac2c4453654f22b2aa780"
server_key_iv: "93e9b00a7b651de5215479eddd8b2767"
server_key_version: "encryptionkey@0"
I20260929 16:29:32.046433 12766 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:32.053535 12766 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:29:32.053864 12766 log.cc:1196] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 16:29:32.054399 12766 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: Deleting consensus metadata
I20260929 16:29:32.054688 12766 master_runner.cc:315] Copying system tablet from 127.5.232.190:35773
I20260929 16:29:32.057323 12766 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: tablet copy: Beginning tablet copy session from remote peer at address 127.5.232.190:35773
I20260929 16:29:32.073611 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.058271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.5.232.190:35773)
0929 16:29:32.058492 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:32.058499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:32.059013 (+ 514us) server_negotiation.cc:407] Connection header received
0929 16:29:32.059186 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:32.059192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:32.059278 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:32.059394 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:32.060822 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.061984 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.063858 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.064520 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.064653 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.067014 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:32.067020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:32.067026 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:32.067064 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:32.070588 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.071443 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.071450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.071454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.071606 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.072054 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.072059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.072063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.072394 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:32.072542 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:32.072996 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 16:29:32.073227 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 16:29:32.073688 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.058215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57360)
0929 16:29:32.058878 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:32.058901 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:32.059055 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:32.059437 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:32.059442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:32.059957 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:32.060650 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.060664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.062158 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.062164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.063701 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.063715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.064704 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.064711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.064898 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.065660 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:32.065689 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:32.066791 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:32.070755 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:32.070761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:32.070766 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:32.071302 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:32.071744 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:32.071749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:32.071754 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:32.071937 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:32.072519 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:32.072525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:32.072733 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 16:29:32.072888 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 16:29:32.073218 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":382,"threads_started":1}
I20260929 16:29:32.074559 12223 tablet_copy_service.cc:140] P c9daca0f91eb486fad1a5699e435f53a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0b43f0470b7c49c593a6dc06d8c12162 ({username='oryx', principal='oryx/127.5.232.188@KRBTEST.COM'} at 127.0.0.1:57360)
I20260929 16:29:32.074720 12223 tablet_copy_service.cc:161] P c9daca0f91eb486fad1a5699e435f53a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0b43f0470b7c49c593a6dc06d8c12162 at {username='oryx', principal='oryx/127.5.232.188@KRBTEST.COM'} at 127.0.0.1:57360: session id = 0b43f0470b7c49c593a6dc06d8c12162-00000000000000000000000000000000
I20260929 16:29:32.078859 12223 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:29:32.080696 12766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:29:32.087683 12766 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: tablet copy: Starting download of 0 data blocks...
I20260929 16:29:32.088003 12766 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: tablet copy: Starting download of 1 WAL segments...
I20260929 16:29:32.091132 12766 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:29:32.095774 12223 tablet_copy_service.cc:342] P c9daca0f91eb486fad1a5699e435f53a: Request end of tablet copy session 0b43f0470b7c49c593a6dc06d8c12162-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.232.188@KRBTEST.COM'} at 127.0.0.1:57360
I20260929 16:29:32.095948 12223 tablet_copy_service.cc:433] P c9daca0f91eb486fad1a5699e435f53a: ending tablet copy session 0b43f0470b7c49c593a6dc06d8c12162-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0b43f0470b7c49c593a6dc06d8c12162
I20260929 16:29:32.115902 12766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:32.127965 12961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:32.129701 12962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:32.143185 12965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:32.143219 12766 server_base.cc:1034] running on GCE node
I20260929 16:29:32.144428 12766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:32.146204 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 12264
I20260929 16:29:32.152245 12766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:32.153468 12766 hybrid_clock.cc:648] HybridClock initialized: now 1790699372153420 us; error 44 us; skew 500 ppm
I20260929 16:29:32.153862 12766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:32 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699372, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:32.156210 12766 init.cc:377] Logged in from keytab as oryx/127.5.232.188@KRBTEST.COM (short username oryx)
I20260929 16:29:32.162364 12766 webserver.cc:460] Webserver started at http://127.5.232.188:42785/ using document root <none> and password file <none>
I20260929 16:29:32.162667 12766 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:32.162778 12766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:32.165146 12766 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 16:29:32.166851 12971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:32.167274 12766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 16:29:32.167413 12766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal
uuid: "0b43f0470b7c49c593a6dc06d8c12162"
format_stamp: "Formatted at 2026-09-29 16:29:31 on dist-test-slave-dssx"
server_key: "2854a5bba41ac2c4453654f22b2aa780"
server_key_iv: "93e9b00a7b651de5215479eddd8b2767"
server_key_version: "encryptionkey@0"
I20260929 16:29:32.167599 12766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:32.170840 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:40865
--local_ip_for_outbound_sockets=127.5.232.129
--tserver_master_addrs=127.5.232.190:35773
--webserver_port=32805
--webserver_interface=127.5.232.129
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab
--principal=oryx/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.232.190:35773,127.5.232.189:42019
--tserver_master_addrs=127.5.232.190:35773,127.5.232.189:42019,127.5.232.188:34743 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:29:32.189661 12766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:32.192901 12766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:32.231319 12766 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.188:34743
I20260929 16:29:32.231375 13023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.188:34743 every 8 connection(s)
I20260929 16:29:32.232460 12766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
I20260929 16:29:32.238425 13024 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:29:32.242621 13024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: Bootstrap starting.
I20260929 16:29:32.260679 13024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:29:32.261173 13024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: Bootstrap complete.
I20260929 16:29:32.262221 13024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } }
I20260929 16:29:32.262516 13024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b43f0470b7c49c593a6dc06d8c12162, State: Initialized, Role: LEARNER
I20260929 16:29:32.262785 13024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } }
I20260929 16:29:32.263638 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.263929 13024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:32.264098 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:29:32.277908 13038 catalog_manager.cc:1259] Loaded cluster ID: fa8e4544e6fb4a42ae8ab152fa0bb3fc
I20260929 16:29:32.278026 13038 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: loading cluster ID for follower catalog manager: success
I20260929 16:29:32.281508 13038 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: acquiring CA information for follower catalog manager: success
I20260929 16:29:32.283887 13038 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:29:32.548014 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:32.548306 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:32.548476 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:32.555891 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:32.556007 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:32.556201 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:32.566749 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab
--principal=oryx/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:40865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=32805
--tserver_master_addrs=127.5.232.190:35773,127.5.232.189:42019,127.5.232.188:34743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:32.568214 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:32.569707 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:32.583330 13043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:32.583330 13044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:32.583535 13046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:32.584753 12976 server_base.cc:1034] running on GCE node
I20260929 16:29:32.585453 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:32.586374 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:32.587574 12976 hybrid_clock.cc:648] HybridClock initialized: now 1790699372587526 us; error 70 us; skew 500 ppm
I20260929 16:29:32.587852 12976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:32 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699372, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:32.590937 12976 init.cc:377] Logged in from keytab as oryx/127.5.232.129@KRBTEST.COM (short username oryx)
I20260929 16:29:32.592442 12976 webserver.cc:460] Webserver started at http://127.5.232.129:32805/ using document root <none> and password file <none>
I20260929 16:29:32.593117 12976 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:32.593232 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:32.599035 12976 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:29:32.601832 13053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:32.603291 12976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:29:32.603442 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "9a026605c1054cf3a71540cc71a19432"
format_stamp: "Formatted at 2026-09-29 16:29:29 on dist-test-slave-dssx"
server_key: "772a3e7a30342f66fa78a00de972f74d"
server_key_iv: "4205c3e37767384a6106f5be10885725"
server_key_version: "encryptionkey@0"
I20260929 16:29:32.604668 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:32.613106 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.605988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:34743 (local address 127.0.0.1:35880)
0929 16:29:32.606898 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:32.606920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:32.607082 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:32.607441 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:32.607446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:32.607472 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:32.607671 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.607684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.608882 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.608889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.610727 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.610742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.612349 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.612355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.612542 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.612579 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:29:32.612644 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:29:32.612784 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":199,"threads_started":1}
I20260929 16:29:32.613899 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.606150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.5.232.188:34743)
0929 16:29:32.606900 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:32.606906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:32.607022 (+ 116us) server_negotiation.cc:407] Connection header received
0929 16:29:32.607213 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:32.607217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:32.607298 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:32.607400 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:32.607811 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.608728 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.610900 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.612124 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.613172 (+ 1048us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.613364 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:32.613440 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 16:29:32.613600 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":386,"threads_started":1}
I20260929 16:29:32.615073 12999 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 16:29:32.615736 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9daca0f91eb486fad1a5699e435f53a. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:32.616047 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:29:32.622809 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:32.627626 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:32.627923 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:32.628914 12976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:32.630378 12976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:32.630501 12976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:32.630628 12976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:32.630714 12976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:32.670409 12976 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:40865
I20260929 16:29:32.670459 13168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:40865 every 8 connection(s)
I20260929 16:29:32.671911 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:32.678814 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 12976
I20260929 16:29:32.679224 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 12398
Sep 29 16:29:32 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699372, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.189@KRBTEST.COM
Sep 29 16:29:32 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699372, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.188@KRBTEST.COM
I20260929 16:29:32.704588 13173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.675153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:40465 (local address 127.5.232.189:42019)
0929 16:29:32.675693 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:32.675701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:32.676762 (+ 1061us) server_negotiation.cc:407] Connection header received
0929 16:29:32.680719 (+ 3957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:32.680728 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:32.680859 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:32.680962 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:32.684108 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.685729 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.689919 (+ 4190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.690590 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.690701 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.696201 (+ 5500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:32.696208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:32.696213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:32.696256 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:32.700138 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.701174 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.701181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.701185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.701328 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.702200 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.702208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.702212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.702595 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:32.702720 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:32.703905 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0929 16:29:32.704149 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":484,"threads_started":1}
Sep 29 16:29:32 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699372, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
Sep 29 16:29:32 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.130@KRBTEST.COM
I20260929 16:29:32.707239 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.675481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.5.232.129:40465)
0929 16:29:32.676556 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:32.676617 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:32.680272 (+ 3655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:32.681738 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:32.681756 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:32.682764 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:32.683842 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.683872 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.686014 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.686023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.689028 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.689049 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.691079 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.691089 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.691433 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.692304 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:32.692334 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:32.695920 (+ 3586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:32.700324 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:32.700335 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:32.700358 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:32.701026 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:32.701530 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:32.701534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:32.701538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:32.702026 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:32.702854 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:32.702863 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:32.703411 (+ 548us) client_negotiation.cc:769] Sending connection context
0929 16:29:32.704903 (+ 1492us) client_negotiation.cc:240] Negotiation successful
0929 16:29:32.705561 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":422,"threads_started":1}
I20260929 16:29:32.709017 13171 heartbeater.cc:347] Connected to a master server at 127.5.232.189:42019
I20260929 16:29:32.709156 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.677901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:56665 (local address 127.5.232.188:34743)
0929 16:29:32.678101 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:32.678107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:32.689739 (+ 11632us) server_negotiation.cc:407] Connection header received
0929 16:29:32.689898 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:32.689904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:32.689985 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:32.690075 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:32.691340 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.692342 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.694793 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.695480 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.695595 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.700753 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:32.700762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:32.700769 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:32.700815 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:32.705432 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.706705 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.706711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.706715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.706864 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.707311 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.707317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.707319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.707665 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:32.707793 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:32.708569 (+ 776us) server_negotiation.cc:299] Negotiation successful
0929 16:29:32.708792 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:29:32.709429 13171 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:32.710409 13176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.677309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:34743 (local address 127.5.232.129:56665)
0929 16:29:32.689611 (+ 12302us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:32.689630 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:32.689776 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:32.690241 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:32.690247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:32.690829 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:32.691166 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.691176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.692529 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.692535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.694598 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.694625 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.696049 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.696062 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.696245 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.697000 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:32.697029 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:32.700514 (+ 3485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:32.705898 (+ 5384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:32.705903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:32.705908 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:32.706553 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:32.707014 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:32.707018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:32.707021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:32.707152 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:32.707876 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:32.707883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:32.708036 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 16:29:32.709433 (+ 1397us) client_negotiation.cc:240] Negotiation successful
0929 16:29:32.709838 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12190,"thread_start_us":10638,"threads_started":1}
I20260929 16:29:32.710480 13171 heartbeater.cc:511] Master 127.5.232.189:42019 requested a full tablet report, sending...
I20260929 16:29:32.713027 12802 ts_manager.cc:194] Registered new tserver with Master: 9a026605c1054cf3a71540cc71a19432 (127.5.232.129:40865)
I20260929 16:29:32.713603 13169 heartbeater.cc:347] Connected to a master server at 127.5.232.188:34743
I20260929 16:29:32.713739 13169 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:32.714247 13169 heartbeater.cc:511] Master 127.5.232.188:34743 requested a full tablet report, sending...
I20260929 16:29:32.715811 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.675475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.129:44531)
0929 16:29:32.687677 (+ 12202us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:32.687712 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:32.687855 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:32.695957 (+ 8102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:32.695963 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:32.696956 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:32.697572 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.697585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.699539 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.699545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.701108 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:32.701121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.702848 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.702856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:32.703071 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.703700 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:32.703726 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:32.706977 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:32.712620 (+ 5643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:32.712626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:32.712630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:32.713138 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:32.713870 (+ 732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:32.713875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:32.713878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:32.714057 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:32.714751 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:32.714759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:32.714907 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 16:29:32.715045 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 16:29:32.715347 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11998,"mutex_wait_us":37,"spinlock_wait_cycles":72576,"thread_start_us":13721,"threads_started":1}
I20260929 16:29:32.716110 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.675526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:44531 (local address 127.5.232.190:35773)
0929 16:29:32.695433 (+ 19907us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:32.695439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:32.695460 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:29:32.695520 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:32.695525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:32.695606 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:32.695727 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:32.697814 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.699380 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.701300 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.702440 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.702580 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.707244 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:32.707254 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:32.707260 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:32.707308 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:32.711981 (+ 4673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.713279 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.713286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.713291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.713478 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.714174 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.714178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.714180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.714551 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:32.714675 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:32.715474 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 16:29:32.715697 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19769,"thread_start_us":19674,"threads_started":1}
I20260929 16:29:32.716398 12989 ts_manager.cc:194] Registered new tserver with Master: 9a026605c1054cf3a71540cc71a19432 (127.5.232.129:40865)
I20260929 16:29:32.717172 13170 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35773
I20260929 16:29:32.717293 13170 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:32.717705 13170 heartbeater.cc:511] Master 127.5.232.190:35773 requested a full tablet report, sending...
I20260929 16:29:32.717842 13179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:32.681273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.5.232.130:37611)
0929 16:29:32.688633 (+ 7360us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:32.688643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:32.689442 (+ 799us) server_negotiation.cc:407] Connection header received
0929 16:29:32.689519 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:32.689530 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:32.689908 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:32.690101 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:32.693508 (+ 3407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.696258 (+ 2750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.700033 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:32.700997 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:32.701138 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:32.707864 (+ 6726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:32.707883 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:32.707901 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:32.707943 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:32.712463 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.714636 (+ 2173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.714645 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.714649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.714812 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:32.715351 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:32.715359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:32.715364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:32.715833 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:32.716469 (+ 636us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:32.717240 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 16:29:32.717549 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7171,"thread_start_us":7025,"threads_started":1}
I20260929 16:29:32.718842 12203 ts_manager.cc:194] Re-registered known tserver with Master: 9a026605c1054cf3a71540cc71a19432 (127.5.232.129:40865)
I20260929 16:29:32.719260 12521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:32.719748 12203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:44531
I20260929 16:29:32.803862 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 12398
I20260929 16:29:32.820361 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:37611
--local_ip_for_outbound_sockets=127.5.232.130
--tserver_master_addrs=127.5.232.190:35773
--webserver_port=35269
--webserver_interface=127.5.232.130
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab
--principal=oryx/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.232.190:35773,127.5.232.189:42019
--tserver_master_addrs=127.5.232.190:35773,127.5.232.189:42019,127.5.232.188:34743 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:29:33.052094 13054 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: NON_VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: true } } }
I20260929 16:29:33.052583 13054 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:33.054009 13182 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: attempting to promote NON_VOTER 0b43f0470b7c49c593a6dc06d8c12162 to VOTER
I20260929 16:29:33.054580 13182 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } }
I20260929 16:29:33.056861 12999 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 1 LEARNER]: Refusing update from remote peer c9daca0f91eb486fad1a5699e435f53a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 16:29:33.056936 12812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Refusing update from remote peer c9daca0f91eb486fad1a5699e435f53a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 16:29:33.057647 13182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 16:29:33.058076 13183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 16:29:33.062345 13054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0b43f0470b7c49c593a6dc06d8c12162 (127.5.232.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:33.066203 12999 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0b43f0470b7c49c593a6dc06d8c12162 (127.5.232.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:33.066766 13054 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:33.067026 13054 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:33.067373 13054 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:33.067476 13182 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:33.067718 13054 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:33.067773 13182 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:33.069862 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:33.070111 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:29:33.069787 12812 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0b43f0470b7c49c593a6dc06d8c12162 (127.5.232.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:33.072907 13188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9daca0f91eb486fad1a5699e435f53a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:33.073321 13188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 16:29:33.158057 13181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:33.158339 13181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:33.158501 13181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:33.165810 13181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:33.165953 13181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:33.166136 13181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:33.176262 13181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab
--principal=oryx/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:37611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=35269
--tserver_master_addrs=127.5.232.190:35773,127.5.232.189:42019,127.5.232.188:34743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:33.177632 13181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:33.179076 13181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:33.192253 13193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:33.192293 13196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:33.192327 13194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:33.193250 13181 server_base.cc:1034] running on GCE node
I20260929 16:29:33.193871 13181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:33.194861 13181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:33.196094 13181 hybrid_clock.cc:648] HybridClock initialized: now 1790699373196032 us; error 43 us; skew 500 ppm
I20260929 16:29:33.196432 13181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:33 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699373, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:33.199681 13181 init.cc:377] Logged in from keytab as oryx/127.5.232.130@KRBTEST.COM (short username oryx)
I20260929 16:29:33.201174 13181 webserver.cc:460] Webserver started at http://127.5.232.130:35269/ using document root <none> and password file <none>
I20260929 16:29:33.201948 13181 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:33.202123 13181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:33.207406 13181 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:29:33.210145 13203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:33.211616 13181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:33.211786 13181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "01ed68c31236410e9aac71d98835d8df"
format_stamp: "Formatted at 2026-09-29 16:29:30 on dist-test-slave-dssx"
server_key: "f79d09724bffe8fed07ec83e93b1b486"
server_key_iv: "a6a46b75f8f7e460dce8cda58dcb2117"
server_key_version: "encryptionkey@0"
I20260929 16:29:33.212805 13181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:33.237519 13181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:33.242733 13181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:33.243041 13181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:33.244203 13181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:33.246006 13181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:33.246119 13181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:33.246218 13181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:33.246289 13181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:33.279999 13181 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:37611
I20260929 16:29:33.280079 13315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:37611 every 8 connection(s)
I20260929 16:29:33.281562 13181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:33.292090 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 13181
I20260929 16:29:33.292394 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 12531
Sep 29 16:29:33 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699373, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
Sep 29 16:29:33 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699373, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.189@KRBTEST.COM
Sep 29 16:29:33 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.131@KRBTEST.COM
Sep 29 16:29:33 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699373, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.188@KRBTEST.COM
I20260929 16:29:33.322412 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.284773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:33587 (local address 127.5.232.190:35773)
0929 16:29:33.291703 (+ 6930us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.291714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.293176 (+ 1462us) server_negotiation.cc:407] Connection header received
0929 16:29:33.296775 (+ 3599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.296784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.296892 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.297030 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.300308 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.301755 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.304757 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.305946 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.306068 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.313480 (+ 7412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.313489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.313494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.313548 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.318074 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.319375 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.319381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.319385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.319531 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.320005 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.320008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.320010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.320390 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.320521 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.321785 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.322010 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6743,"thread_start_us":6605,"threads_started":1}
I20260929 16:29:33.323508 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.285161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.5.232.130:43573)
0929 16:29:33.297166 (+ 12005us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:33.297186 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:33.297357 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:33.298035 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:33.298040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:33.298843 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:33.299912 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.299948 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.305138 (+ 5190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.305141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.307034 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.307048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.308015 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.308020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.308134 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.309068 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:33.309090 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:33.313682 (+ 4592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:33.318165 (+ 4483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.318176 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.318197 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.319012 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.319490 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.319497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.319501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.319736 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.320567 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:33.320577 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:33.321086 (+ 509us) client_negotiation.cc:769] Sending connection context
0929 16:29:33.321412 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 16:29:33.321995 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11838,"mutex_wait_us":123,"spinlock_wait_cycles":64256,"thread_start_us":12170,"threads_started":1}
I20260929 16:29:33.324649 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.299842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:34743 (local address 127.5.232.130:42515)
0929 16:29:33.303379 (+ 3537us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:33.303418 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:33.303541 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:33.304057 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:33.304061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:33.304734 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:33.305721 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.305734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.309206 (+ 3472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.309215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.312171 (+ 2956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.312219 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.314008 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.314016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.314231 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.314936 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:33.314953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:33.317682 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:33.321709 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.321714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.321717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.322202 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.322732 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.322742 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.322745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.322891 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.323524 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:33.323529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:33.323808 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 16:29:33.323961 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 16:29:33.324262 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3404,"thread_start_us":3320,"threads_started":1}
I20260929 16:29:33.324680 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.284491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:43573 (local address 127.5.232.189:42019)
0929 16:29:33.292136 (+ 7645us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.292145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.297308 (+ 5163us) server_negotiation.cc:407] Connection header received
0929 16:29:33.297526 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.297532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.297629 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.297787 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.303464 (+ 5677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.304598 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.307178 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.307827 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.307982 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.313915 (+ 5933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.313922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.313929 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.313986 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.317969 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.319171 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.319180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.319184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.319337 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.319938 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.319945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.319950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.320328 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.320490 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.323499 (+ 3009us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.323912 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7491,"thread_start_us":7377,"threads_started":1}
I20260929 16:29:33.325590 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.298087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:42515 (local address 127.5.232.188:34743)
0929 16:29:33.303143 (+ 5056us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.303149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.303639 (+ 490us) server_negotiation.cc:407] Connection header received
0929 16:29:33.303705 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.303710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.303782 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.303879 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.305919 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.309030 (+ 3111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.312446 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.313550 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.313695 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.317997 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.318004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.318008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.318046 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.321569 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.322346 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.322352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.322355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.322510 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.323020 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.323025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.323028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.323319 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.323457 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.324993 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.325207 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4911,"thread_start_us":4767,"threads_started":1}
I20260929 16:29:33.325829 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.285161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.130:33587)
0929 16:29:33.293000 (+ 7839us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:33.293071 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:33.296337 (+ 3266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:33.297435 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:33.297453 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:33.298628 (+ 1175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:33.300146 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.300159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.302032 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.302040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.304582 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.304599 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.306281 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.306287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.306495 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.307381 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:33.307429 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:33.310871 (+ 3442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:33.318243 (+ 7372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.318248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.318255 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.319219 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.319666 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.319673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.319677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.319863 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.320595 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:33.320602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:33.321103 (+ 501us) client_negotiation.cc:769] Sending connection context
0929 16:29:33.321412 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 16:29:33.325440 (+ 4028us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7478,"mutex_wait_us":4286,"spinlock_wait_cycles":44160,"thread_start_us":7244,"threads_started":1}
I20260929 16:29:33.325973 13318 heartbeater.cc:347] Connected to a master server at 127.5.232.189:42019
I20260929 16:29:33.326236 13318 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:33.326419 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.293705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.5.232.131:39939)
0929 16:29:33.299580 (+ 5875us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.299591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.299626 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:29:33.299698 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.299709 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.299987 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.302925 (+ 2938us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.303169 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.304726 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.308177 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.309095 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.309237 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.317344 (+ 8107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.317365 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.317422 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.317478 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.321589 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.322420 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.322428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.322432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.322917 (+ 485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.323359 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.323368 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.323374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.323790 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.324099 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.325354 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.325644 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5702,"thread_start_us":5558,"threads_started":1}
I20260929 16:29:33.326870 13317 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35773
I20260929 16:29:33.327003 13317 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:33.327489 13316 heartbeater.cc:347] Connected to a master server at 127.5.232.188:34743
I20260929 16:29:33.327632 13316 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:33.327704 12654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:33.328132 13318 heartbeater.cc:511] Master 127.5.232.189:42019 requested a full tablet report, sending...
I20260929 16:29:33.328166 13317 heartbeater.cc:511] Master 127.5.232.190:35773 requested a full tablet report, sending...
I20260929 16:29:33.328459 13316 heartbeater.cc:511] Master 127.5.232.188:34743 requested a full tablet report, sending...
I20260929 16:29:33.329850 12203 ts_manager.cc:194] Re-registered known tserver with Master: 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611)
I20260929 16:29:33.330032 12802 ts_manager.cc:194] Registered new tserver with Master: 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611)
I20260929 16:29:33.331295 12203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:33587
I20260929 16:29:33.332432 12989 ts_manager.cc:194] Registered new tserver with Master: 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611)
I20260929 16:29:33.418715 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 12531
I20260929 16:29:33.434947 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:39939
--local_ip_for_outbound_sockets=127.5.232.131
--tserver_master_addrs=127.5.232.190:35773
--webserver_port=40457
--webserver_interface=127.5.232.131
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab
--principal=oryx/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.232.190:35773,127.5.232.189:42019
--tserver_master_addrs=127.5.232.190:35773,127.5.232.189:42019,127.5.232.188:34743 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:29:33.722676 13170 heartbeater.cc:503] Master 127.5.232.190:35773 was elected leader, sending a full tablet report...
W20260929 16:29:33.779714 13329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:33.780048 13329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:33.780215 13329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:33.787824 13329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:33.787992 13329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:33.788166 13329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:33.798473 13329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab
--principal=oryx/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:39939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=40457
--tserver_master_addrs=127.5.232.190:35773,127.5.232.189:42019,127.5.232.188:34743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:33.799880 13329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:33.801259 13329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:33.815699 13335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:33.815732 13337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:33.815780 13334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:33.816179 13329 server_base.cc:1034] running on GCE node
I20260929 16:29:33.817087 13329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:33.818149 13329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:33.819365 13329 hybrid_clock.cc:648] HybridClock initialized: now 1790699373819306 us; error 54 us; skew 500 ppm
I20260929 16:29:33.819706 13329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:33 dist-test-slave-dssx krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699373, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:33.823030 13329 init.cc:377] Logged in from keytab as oryx/127.5.232.131@KRBTEST.COM (short username oryx)
I20260929 16:29:33.824707 13329 webserver.cc:460] Webserver started at http://127.5.232.131:40457/ using document root <none> and password file <none>
I20260929 16:29:33.825554 13329 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:33.825713 13329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:33.831146 13329 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:29:33.834020 13344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:33.835449 13329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:33.835630 13329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "c28cf18f75454f029b1693e7ec5bf493"
format_stamp: "Formatted at 2026-09-29 16:29:30 on dist-test-slave-dssx"
server_key: "012ceb5948e86d9f82bf72de617354dd"
server_key_iv: "a53d128d485f54a48b2149b1893fae54"
server_key_version: "encryptionkey@0"
I20260929 16:29:33.836637 13329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:33.865324 13329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:33.870136 13329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:33.870431 13329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:33.871428 13329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:33.872869 13329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:33.872985 13329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:33.873121 13329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:33.873208 13329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:33.906939 13329 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:39939
I20260929 16:29:33.906998 13456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:39939 every 8 connection(s)
I20260929 16:29:33.908475 13329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:33.916750 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 13329
Sep 29 16:29:33 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699373, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:33.971278 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.933937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.5.232.188:34743)
0929 16:29:33.950997 (+ 17060us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.951401 (+ 404us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.951430 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:29:33.951497 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.951502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.951594 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.954901 (+ 3307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.955098 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.956178 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.958118 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.958949 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.959073 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.962725 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.962732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.962736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.962777 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.966937 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.967989 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.967996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.968000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.968147 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.968651 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.968656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.968659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.969096 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.970479 (+ 1383us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.970632 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.970888 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16912,"thread_start_us":17729,"threads_started":1}
Sep 29 16:29:33 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699373, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.189@KRBTEST.COM
I20260929 16:29:33.975914 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.922248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.5.232.189:42019)
0929 16:29:33.938500 (+ 16252us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.938509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.938536 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:29:33.938623 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.938632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.938735 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.938905 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.942502 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.943916 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.946377 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.947607 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.947738 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.963814 (+ 16076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.963824 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.963829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.963883 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.968691 (+ 4808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.970558 (+ 1867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.970565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.970568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.970741 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.971212 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.971217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.971220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.971857 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.972100 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.975237 (+ 3137us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.975510 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16088,"thread_start_us":16808,"threads_started":1}
I20260929 16:29:33.976603 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.911612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:49277 (local address 127.5.232.190:35773)
0929 16:29:33.935007 (+ 23395us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.935016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.936012 (+ 996us) server_negotiation.cc:407] Connection header received
0929 16:29:33.941442 (+ 5430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.941450 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.941559 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.941660 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.950868 (+ 9208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.952343 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.957739 (+ 5396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.958912 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.959017 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.967056 (+ 8039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.967063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.967067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.967107 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.970798 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.973222 (+ 2424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.973230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.973236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.973452 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.974274 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.974279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.974282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.974671 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.974824 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.975725 (+ 901us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.975950 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23205,"thread_start_us":36620,"threads_started":1}
I20260929 16:29:33.977525 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.912504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.131:49277)
0929 16:29:33.935799 (+ 23295us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:33.935866 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:33.941121 (+ 5255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:33.942984 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:33.943003 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:33.945346 (+ 2343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:33.946395 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.946425 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.954733 (+ 8308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.954741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.957582 (+ 2841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.957594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.959355 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.959363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.959665 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.960400 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:33.960434 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:33.964976 (+ 4542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:33.972372 (+ 7396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.972382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.972403 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.973061 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.973915 (+ 854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.973919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.973923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.974159 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.974824 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:33.974832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:33.975290 (+ 458us) client_negotiation.cc:769] Sending connection context
0929 16:29:33.975600 (+ 310us) client_negotiation.cc:240] Negotiation successful
0929 16:29:33.976126 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22954,"thread_start_us":22680,"threads_started":1}
Sep 29 16:29:33 dist-test-slave-dssx krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699373, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.188@KRBTEST.COM
I20260929 16:29:33.979488 13458 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35773
I20260929 16:29:33.979914 13458 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:33.980870 13458 heartbeater.cc:511] Master 127.5.232.190:35773 requested a full tablet report, sending...
I20260929 16:29:33.983047 12203 ts_manager.cc:194] Re-registered known tserver with Master: c28cf18f75454f029b1693e7ec5bf493 (127.5.232.131:39939)
I20260929 16:29:33.983734 13461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.921173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.5.232.190:35773)
0929 16:29:33.947923 (+ 26750us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.947932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.947958 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:29:33.948021 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.948029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.948116 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.950645 (+ 2529us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.951735 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.952884 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.957254 (+ 4370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.957960 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.963215 (+ 5255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.968956 (+ 5741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.968964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.968969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.969009 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.979712 (+ 10703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.980613 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.980619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.980623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.980748 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.981216 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.981221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.981224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.981645 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.982941 (+ 1296us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.983075 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.983303 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26563,"thread_start_us":13858,"threads_started":1}
I20260929 16:29:33.983860 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.943120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.5.232.131:50197)
0929 16:29:33.950696 (+ 7576us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:33.950717 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:33.950834 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:33.954609 (+ 3775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:33.954616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:33.955163 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:33.955541 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.955557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.962469 (+ 6912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.962477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.965275 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.965286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.966518 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.966524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.966702 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.971029 (+ 4327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:33.971053 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:33.973763 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:33.980763 (+ 7000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.980768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.980772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.981424 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.981859 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.981866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.981870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.982029 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.982873 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:33.982877 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:33.983033 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 16:29:33.983162 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:29:33.983464 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7461,"thread_start_us":7786,"threads_started":1}
I20260929 16:29:33.984179 12203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:49277
I20260929 16:29:33.984771 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.937028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:50197 (local address 127.5.232.189:42019)
0929 16:29:33.954138 (+ 17110us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.954144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.954161 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:29:33.954216 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.954221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.954296 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.954392 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.961235 (+ 6843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.962318 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.965460 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.966136 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.966258 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.974002 (+ 7744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.974012 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.974020 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.974067 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.980590 (+ 6523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.981572 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.981578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.981581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.981717 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.982159 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.982165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.982168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.982518 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.982666 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.984110 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.984350 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16945,"thread_start_us":23995,"threads_started":1}
I20260929 16:29:33.985677 13459 heartbeater.cc:347] Connected to a master server at 127.5.232.189:42019
I20260929 16:29:33.985821 13459 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:33.986768 13459 heartbeater.cc:511] Master 127.5.232.189:42019 requested a full tablet report, sending...
I20260929 16:29:33.988024 12802 ts_manager.cc:194] Registered new tserver with Master: c28cf18f75454f029b1693e7ec5bf493 (127.5.232.131:39939)
I20260929 16:29:33.989598 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.913172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:34743 (local address 127.5.232.131:56513)
0929 16:29:33.936658 (+ 23486us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:33.936675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:33.940360 (+ 3685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:33.954300 (+ 13940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:33.954304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:33.955763 (+ 1459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:33.956036 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.956047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.957608 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.957613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.959470 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:33.959481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.960594 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.960603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:33.960798 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.962605 (+ 1807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:33.962631 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:33.980045 (+ 17414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:33.984469 (+ 4424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.984474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.984478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.984968 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.985592 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:33.985596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:33.985600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:33.986676 (+ 1076us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:33.987606 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:33.987610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:33.988807 (+ 1197us) client_negotiation.cc:769] Sending connection context
0929 16:29:33.988975 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 16:29:33.989220 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23357,"mutex_wait_us":3,"thread_start_us":23213,"threads_started":1}
I20260929 16:29:33.989837 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:33.911779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:56513 (local address 127.5.232.188:34743)
0929 16:29:33.915367 (+ 3588us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:33.915372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:33.936759 (+ 21387us) server_negotiation.cc:407] Connection header received
0929 16:29:33.953130 (+ 16371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:33.953139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:33.953237 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:33.953800 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:33.956221 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.957430 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.959619 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:33.960217 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:33.960323 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:33.980312 (+ 19989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:33.980320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:33.980325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:33.980369 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:33.984313 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.985236 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.985241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.985245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.985446 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:33.987012 (+ 1566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:33.987017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:33.987020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:33.987388 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:33.987512 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:33.989115 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0929 16:29:33.989429 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3441,"thread_start_us":3334,"threads_started":1}
I20260929 16:29:33.990592 13457 heartbeater.cc:347] Connected to a master server at 127.5.232.188:34743
I20260929 16:29:33.990732 13457 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:33.991098 13457 heartbeater.cc:511] Master 127.5.232.188:34743 requested a full tablet report, sending...
I20260929 16:29:33.992113 12989 ts_manager.cc:194] Registered new tserver with Master: c28cf18f75454f029b1693e7ec5bf493 (127.5.232.131:39939)
I20260929 16:29:33.989429 12202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:33.997256 12202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:34.030202 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.018895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.5.232.131:39939)
0929 16:29:34.019949 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.019958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.019986 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:29:34.020073 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.020079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.020375 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.020601 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:34.021661 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.023228 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.027236 (+ 4008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.028421 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.028561 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.028782 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.029665 (+ 883us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.029837 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":742,"threads_started":1}
I20260929 16:29:34.030450 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.017184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:37611 (local address 127.0.0.1:33780)
0929 16:29:34.017869 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:34.017890 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:34.018014 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:34.023690 (+ 5676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:34.023695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:34.023716 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:34.024285 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.024297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.026114 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.026118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.027908 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.027922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.029600 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.029606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.029777 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.029806 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:29:34.029900 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:29:34.030041 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":52,"thread_start_us":363,"threads_started":1}
I20260929 16:29:34.030895 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.018004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40865 (local address 127.0.0.1:32916)
0929 16:29:34.018514 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:34.018532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:34.018635 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:34.022513 (+ 3878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:34.022521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:34.022557 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:34.022897 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.022907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.025185 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.025190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.027076 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.027087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.030026 (+ 2939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.030032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.030221 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.030261 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:29:34.030365 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 16:29:34.030490 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"spinlock_wait_cycles":3968,"thread_start_us":353,"threads_started":1}
I20260929 16:29:34.030907 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.018258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.5.232.129:40865)
0929 16:29:34.021863 (+ 3605us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.021873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.021902 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:29:34.022016 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.022026 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.022296 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.022497 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:34.023089 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.025033 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.027218 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.029806 (+ 2588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.029926 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.030109 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.030450 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.030621 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3429,"thread_start_us":3307,"threads_started":1}
I20260929 16:29:34.031009 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.018648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:39939 (local address 127.0.0.1:56376)
0929 16:29:34.019359 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:34.019381 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:34.019515 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:34.020717 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:34.020725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:34.020759 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:34.021412 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.021427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.023372 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.023378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.025598 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.025615 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.028636 (+ 3021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.028642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.029249 (+ 607us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.029290 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:29:34.029380 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:29:34.030678 (+ 1298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":1204,"threads_started":1}
I20260929 16:29:34.032704 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.017466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.5.232.130:37611)
0929 16:29:34.023009 (+ 5543us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.023018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.023043 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:29:34.023139 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.023147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.023403 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.023597 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:34.024510 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.026002 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.028237 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.029376 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.029643 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.029828 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.032392 (+ 2564us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.032525 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5385,"thread_start_us":5267,"threads_started":1}
I20260929 16:29:34.035145 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 3cad8e22848543869b35f899a7ee9835 (DEFAULT_TABLE table=test-table [id=cb6b896f254947a79d0f52e1505d423c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:34.035147 13392 tablet_service.cc:1467] Processing CreateTablet for tablet 3cad8e22848543869b35f899a7ee9835 (DEFAULT_TABLE table=test-table [id=cb6b896f254947a79d0f52e1505d423c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:34.036639 13392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cad8e22848543869b35f899a7ee9835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:34.036639 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cad8e22848543869b35f899a7ee9835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:34.036334 13251 tablet_service.cc:1467] Processing CreateTablet for tablet 3cad8e22848543869b35f899a7ee9835 (DEFAULT_TABLE table=test-table [id=cb6b896f254947a79d0f52e1505d423c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:34.037356 13251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cad8e22848543869b35f899a7ee9835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:34.050494 13491 tablet_bootstrap.cc:492] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: Bootstrap starting.
I20260929 16:29:34.053689 13492 tablet_bootstrap.cc:492] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: Bootstrap starting.
I20260929 16:29:34.056066 13491 tablet_bootstrap.cc:654] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:34.057227 13492 tablet_bootstrap.cc:654] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:34.057675 13491 log.cc:824] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:34.058364 13493 tablet_bootstrap.cc:492] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: Bootstrap starting.
I20260929 16:29:34.058584 13492 log.cc:824] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:34.061858 13493 tablet_bootstrap.cc:654] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:34.062403 13491 tablet_bootstrap.cc:492] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: No bootstrap required, opened a new log
I20260929 16:29:34.062702 13491 ts_tablet_manager.cc:1397] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260929 16:29:34.063117 13493 log.cc:824] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:34.063158 13492 tablet_bootstrap.cc:492] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: No bootstrap required, opened a new log
I20260929 16:29:34.063438 13492 ts_tablet_manager.cc:1397] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 16:29:34.065949 13493 tablet_bootstrap.cc:492] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: No bootstrap required, opened a new log
I20260929 16:29:34.066214 13493 ts_tablet_manager.cc:1397] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 16:29:34.073836 13491 raft_consensus.cc:348] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } }
I20260929 16:29:34.074059 13492 raft_consensus.cc:348] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } }
I20260929 16:29:34.074331 13491 raft_consensus.cc:374] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:34.074430 13492 raft_consensus.cc:374] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:34.074427 13491 raft_consensus.cc:729] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ed68c31236410e9aac71d98835d8df, State: Initialized, Role: FOLLOWER
I20260929 16:29:34.074530 13492 raft_consensus.cc:729] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a026605c1054cf3a71540cc71a19432, State: Initialized, Role: FOLLOWER
I20260929 16:29:34.074211 13493 raft_consensus.cc:348] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } }
I20260929 16:29:34.074707 13493 raft_consensus.cc:374] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:34.074836 13493 raft_consensus.cc:729] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28cf18f75454f029b1693e7ec5bf493, State: Initialized, Role: FOLLOWER
I20260929 16:29:34.075120 13492 consensus_queue.cc:260] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } }
I20260929 16:29:34.075120 13491 consensus_queue.cc:260] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } }
I20260929 16:29:34.075884 13493 consensus_queue.cc:260] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } }
I20260929 16:29:34.077749 13493 ts_tablet_manager.cc:1428] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 16:29:34.078014 13458 heartbeater.cc:503] Master 127.5.232.190:35773 was elected leader, sending a full tablet report...
I20260929 16:29:34.083571 13492 ts_tablet_manager.cc:1428] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: Time spent starting tablet: real 0.020s user 0.010s sys 0.010s
I20260929 16:29:34.084537 13317 heartbeater.cc:503] Master 127.5.232.190:35773 was elected leader, sending a full tablet report...
I20260929 16:29:34.086376 13491 ts_tablet_manager.cc:1428] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: Time spent starting tablet: real 0.024s user 0.015s sys 0.008s
I20260929 16:29:34.092101 13499 raft_consensus.cc:482] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:34.092442 13499 raft_consensus.cc:504] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } }
I20260929 16:29:34.094166 13499 leader_election.cc:288] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611), 9a026605c1054cf3a71540cc71a19432 (127.5.232.129:40865)
I20260929 16:29:34.100251 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.094762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40865 (local address 127.5.232.131:56889)
0929 16:29:34.094891 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:34.094913 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:34.094992 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:34.095384 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:34.095389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:34.095411 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:34.095614 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.095628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.096602 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.096607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.098339 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.098352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.099453 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.099459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.099641 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.099666 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:29:34.099775 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:29:34.099912 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":62}
I20260929 16:29:34.100543 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.094493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:48513 (local address 127.5.232.130:37611)
0929 16:29:34.094658 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.094666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.094686 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:29:34.094924 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.094930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.095060 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.095166 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:34.095536 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.096338 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.098475 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.099661 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.099830 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.099900 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.100173 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.100298 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:29:34.100567 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.094416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:37611 (local address 127.5.232.131:48513)
0929 16:29:34.094619 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:34.094639 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:34.094758 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:34.095200 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:34.095204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:34.095229 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:34.095387 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.095394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.096494 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.096500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.098317 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:34.098330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.099822 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.099828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:34.100018 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.100041 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:29:34.100149 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:29:34.100284 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":25}
I20260929 16:29:34.101285 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.094846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:56889 (local address 127.5.232.129:40865)
0929 16:29:34.095161 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.095165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.095179 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:29:34.095222 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.095225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.095271 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.095388 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:34.095734 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.096470 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.098496 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.099264 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.099453 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.099531 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.100876 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.101024 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260929 16:29:34.101866 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cad8e22848543869b35f899a7ee9835" candidate_uuid: "c28cf18f75454f029b1693e7ec5bf493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ed68c31236410e9aac71d98835d8df" is_pre_election: true
I20260929 16:29:34.102305 13271 raft_consensus.cc:2454] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28cf18f75454f029b1693e7ec5bf493 in term 0.
I20260929 16:29:34.102177 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cad8e22848543869b35f899a7ee9835" candidate_uuid: "c28cf18f75454f029b1693e7ec5bf493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a026605c1054cf3a71540cc71a19432" is_pre_election: true
I20260929 16:29:34.102689 13124 raft_consensus.cc:2454] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28cf18f75454f029b1693e7ec5bf493 in term 0.
I20260929 16:29:34.102975 13345 leader_election.cc:302] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ed68c31236410e9aac71d98835d8df, c28cf18f75454f029b1693e7ec5bf493; no voters:
I20260929 16:29:34.103358 13499 raft_consensus.cc:2781] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:34.103497 13499 raft_consensus.cc:482] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:34.103598 13499 raft_consensus.cc:3037] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:34.105931 13499 raft_consensus.cc:504] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } }
I20260929 16:29:34.106617 13499 leader_election.cc:288] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [CANDIDATE]: Term 1 election: Requested vote from peers 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611), 9a026605c1054cf3a71540cc71a19432 (127.5.232.129:40865)
I20260929 16:29:34.107128 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cad8e22848543869b35f899a7ee9835" candidate_uuid: "c28cf18f75454f029b1693e7ec5bf493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ed68c31236410e9aac71d98835d8df"
I20260929 16:29:34.107234 13124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cad8e22848543869b35f899a7ee9835" candidate_uuid: "c28cf18f75454f029b1693e7ec5bf493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a026605c1054cf3a71540cc71a19432"
I20260929 16:29:34.107383 13271 raft_consensus.cc:3037] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:34.107453 13124 raft_consensus.cc:3037] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:34.109747 13124 raft_consensus.cc:2454] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28cf18f75454f029b1693e7ec5bf493 in term 1.
I20260929 16:29:34.109946 13271 raft_consensus.cc:2454] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28cf18f75454f029b1693e7ec5bf493 in term 1.
I20260929 16:29:34.110278 13345 leader_election.cc:302] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a026605c1054cf3a71540cc71a19432, c28cf18f75454f029b1693e7ec5bf493; no voters:
I20260929 16:29:34.110600 13499 raft_consensus.cc:2781] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:34.111107 13499 raft_consensus.cc:686] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 1 LEADER]: Becoming Leader. State: Replica: c28cf18f75454f029b1693e7ec5bf493, State: Running, Role: LEADER
I20260929 16:29:34.111651 13499 consensus_queue.cc:237] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } }
I20260929 16:29:34.117010 12203 catalog_manager.cc:5495] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 reported cstate change: term changed from 0 to 1, leader changed from <none> to c28cf18f75454f029b1693e7ec5bf493 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "c28cf18f75454f029b1693e7ec5bf493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } health_report { overall_health: HEALTHY } } }
I20260929 16:29:34.157202 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.152287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.5.232.131:39939)
0929 16:29:34.152466 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.152471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.152489 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:29:34.152686 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.152690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.152767 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.152900 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:34.153212 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.153983 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.155257 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.155653 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.155748 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.156189 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:34.156712 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:34.156835 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.156915 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.156989 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:29:34.166222 13271 raft_consensus.cc:1261] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [term 1 FOLLOWER]: Refusing update from remote peer c28cf18f75454f029b1693e7ec5bf493: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:34.166242 13124 raft_consensus.cc:1261] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [term 1 FOLLOWER]: Refusing update from remote peer c28cf18f75454f029b1693e7ec5bf493: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:34.167222 13500 consensus_queue.cc:1035] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:34.167626 13499 consensus_queue.cc:1035] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:29:34.168361 13460 tablet.cc:2369] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:29:34.176433 13172 tablet.cc:2369] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:29:34.203704 12203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:34.204016 12203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:34.213922 12203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9daca0f91eb486fad1a5699e435f53a: Sending DeleteTablet for 3 replicas of tablet 3cad8e22848543869b35f899a7ee9835
I20260929 16:29:34.215224 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cad8e22848543869b35f899a7ee9835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:34 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:32916
I20260929 16:29:34.215287 13392 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cad8e22848543869b35f899a7ee9835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:34 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:56376
I20260929 16:29:34.216619 13251 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cad8e22848543869b35f899a7ee9835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:34 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:33780
I20260929 16:29:34.216806 13512 tablet_replica.cc:330] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: stopping tablet replica
I20260929 16:29:34.216567 13250 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cad8e22848543869b35f899a7ee9835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:34 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:33780
I20260929 16:29:34.217340 13512 raft_consensus.cc:2229] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:29:34.218035 13512 raft_consensus.cc:2258] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:34.220860 13512 ts_tablet_manager.cc:1905] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:34.229976 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 12171
I20260929 16:29:34.230250 13512 ts_tablet_manager.cc:1918] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:34.230448 13512 log.cc:1196] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/3cad8e22848543869b35f899a7ee9835
I20260929 16:29:34.231060 13512 ts_tablet_manager.cc:1939] T 3cad8e22848543869b35f899a7ee9835 P c28cf18f75454f029b1693e7ec5bf493: Deleting consensus metadata
I20260929 16:29:34.231211 12233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:34.242637 13514 tablet_replica.cc:330] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: stopping tablet replica
I20260929 16:29:34.243274 13513 tablet_replica.cc:330] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: stopping tablet replica
I20260929 16:29:34.243839 13514 raft_consensus.cc:2229] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:34.244351 13513 raft_consensus.cc:2229] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:34.245018 13513 raft_consensus.cc:2258] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:34.245532 13514 raft_consensus.cc:2258] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:34.248979 13513 ts_tablet_manager.cc:1905] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:34.252323 13514 ts_tablet_manager.cc:1905] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:34.257371 13513 ts_tablet_manager.cc:1918] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:34.257699 13513 log.cc:1196] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/3cad8e22848543869b35f899a7ee9835
I20260929 16:29:34.258270 13513 ts_tablet_manager.cc:1939] T 3cad8e22848543869b35f899a7ee9835 P 9a026605c1054cf3a71540cc71a19432: Deleting consensus metadata
I20260929 16:29:34.260097 13514 ts_tablet_manager.cc:1918] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:34.260257 13514 log.cc:1196] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/3cad8e22848543869b35f899a7ee9835
I20260929 16:29:34.260744 13514 ts_tablet_manager.cc:1939] T 3cad8e22848543869b35f899a7ee9835 P 01ed68c31236410e9aac71d98835d8df: Deleting consensus metadata
I20260929 16:29:34.265177 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.218516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.5.232.189:42019)
0929 16:29:34.218686 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.218692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.228080 (+ 9388us) server_negotiation.cc:407] Connection header received
0929 16:29:34.233860 (+ 5780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.233869 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.233991 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.234090 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:34.238120 (+ 4030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.239288 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.241642 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.242341 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.242447 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.246065 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:34.246072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:34.246076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:34.246117 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:34.250585 (+ 4468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:34.262460 (+ 11875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:34.262468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:34.262472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:34.262653 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:34.263158 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:34.263164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:34.263167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:34.263541 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:34.263799 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.264498 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.264756 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:29:34.286223 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.228792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.5.232.188:34743)
0929 16:29:34.229000 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.229007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.267528 (+ 38521us) server_negotiation.cc:407] Connection header received
0929 16:29:34.267839 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.267846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.267960 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.268065 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:34.270063 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.271168 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.273333 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.274016 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.274142 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.277735 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:34.277742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:34.277747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:34.277791 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:34.282026 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:34.282940 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:34.282947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:34.282951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:34.283119 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:34.283583 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:34.283589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:34.283593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:34.283947 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:34.284183 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.285646 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.285887 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:29:34.417897 12187 catalog_manager.cc:4841] TS 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611): tablet 3cad8e22848543869b35f899a7ee9835 (table test-table [id=cb6b896f254947a79d0f52e1505d423c]) successfully deleted
I20260929 16:29:34.418581 12189 catalog_manager.cc:4841] TS c28cf18f75454f029b1693e7ec5bf493 (127.5.232.131:39939): tablet 3cad8e22848543869b35f899a7ee9835 (table test-table [id=cb6b896f254947a79d0f52e1505d423c]) successfully deleted
W20260929 16:29:34.418792 12189 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.5.232.131:39939, user_credentials={real_user=slave}} blocked reactor thread for 185581us
W20260929 16:29:34.418937 12187 catalog_manager.cc:4808] TS 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611): delete failed for tablet 3cad8e22848543869b35f899a7ee9835 because the tablet was not found. No further retry: Not found: Tablet not found: 3cad8e22848543869b35f899a7ee9835
I20260929 16:29:34.419210 12187 catalog_manager.cc:4841] TS 9a026605c1054cf3a71540cc71a19432 (127.5.232.129:40865): tablet 3cad8e22848543869b35f899a7ee9835 (table test-table [id=cb6b896f254947a79d0f52e1505d423c]) successfully deleted
I20260929 16:29:34.419472 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 12171
I20260929 16:29:34.473978 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.455200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.5.232.188:34743)
0929 16:29:34.455427 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.455436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.455460 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:29:34.455538 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.455546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.455708 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.455843 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:34.459745 (+ 3902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.460564 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.462032 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.462681 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.462805 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.466394 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:34.466400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:34.466405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:34.466469 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:34.469973 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:34.471135 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:34.471144 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:34.471147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:34.471304 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:34.471769 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:34.471778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:34.471785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:34.472178 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:34.472440 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.473315 (+ 875us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.473580 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:29:34.474583 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:34.453854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.5.232.189:42019)
0929 16:29:34.454086 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:34.454095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:34.455120 (+ 1025us) server_negotiation.cc:407] Connection header received
0929 16:29:34.455413 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:34.455419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:34.455500 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:34.455559 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:34.458229 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.459453 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.461004 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:34.461676 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:34.461804 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:34.466394 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:34.466400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:34.466405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:34.466467 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:34.471063 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:34.471877 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:34.471882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:34.471885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:34.472084 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:34.472548 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:34.472553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:34.472556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:34.472928 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:34.473215 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:34.473866 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 16:29:34.474105 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:29:35.194336 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.193547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.131:45521)
0929 16:29:35.194010 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.194137 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":238,"threads_started":1}
W20260929 16:29:35.194777 13458 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:35773 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
I20260929 16:29:35.200394 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.199468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.129:56661)
0929 16:29:35.200031 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.200173 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":258,"threads_started":1}
W20260929 16:29:35.200920 13170 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:35773 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
I20260929 16:29:35.216504 13526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.215763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.130:46887)
0929 16:29:35.216169 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.216284 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":208,"threads_started":1}
W20260929 16:29:35.216948 13317 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:35773 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
I20260929 16:29:35.713737 13527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9daca0f91eb486fad1a5699e435f53a)
I20260929 16:29:35.713829 13528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c9daca0f91eb486fad1a5699e435f53a)
I20260929 16:29:35.714066 13527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } }
I20260929 16:29:35.714100 13528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } }
I20260929 16:29:35.715494 13528 leader_election.cc:288] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9daca0f91eb486fad1a5699e435f53a (127.5.232.190:35773), 0b43f0470b7c49c593a6dc06d8c12162 (127.5.232.188:34743)
I20260929 16:29:35.715529 13527 leader_election.cc:288] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c9daca0f91eb486fad1a5699e435f53a (127.5.232.190:35773), 6721b56a80e64285978fa1b321a9f761 (127.5.232.189:42019)
I20260929 16:29:35.716667 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.715607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57108)
0929 16:29:35.716135 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.716302 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":233,"threads_started":1}
I20260929 16:29:35.716813 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.715629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57118)
0929 16:29:35.716421 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.716584 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":538,"threads_started":1}
W20260929 16:29:35.716940 12786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
W20260929 16:29:35.717087 12972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
I20260929 16:29:35.727780 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.716127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:34743 (local address 127.0.0.1:48124)
0929 16:29:35.717020 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.717039 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:35.717188 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:35.719126 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:35.719138 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:35.719181 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:35.719681 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.719702 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.721735 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.721744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.724345 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.724364 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.726583 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.726593 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.726874 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.726934 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 16:29:35.727019 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:29:35.727222 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":32,"thread_start_us":446,"threads_started":1}
I20260929 16:29:35.728763 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.719860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57132)
0929 16:29:35.728308 (+ 8448us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.728609 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8104,"threads_started":1}
I20260929 16:29:35.728971 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.718014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.5.232.188:34743)
0929 16:29:35.718775 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:35.718781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:35.718798 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:29:35.718871 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:35.718876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:35.718941 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:35.719870 (+ 929us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:35.720076 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.721552 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.724560 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.726347 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.728167 (+ 1820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.728295 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:35.728355 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 16:29:35.728510 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":398,"threads_started":1}
I20260929 16:29:35.729803 12999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6721b56a80e64285978fa1b321a9f761" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b43f0470b7c49c593a6dc06d8c12162" is_pre_election: true
I20260929 16:29:35.730228 12999 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6721b56a80e64285978fa1b321a9f761 in term 1.
W20260929 16:29:35.731729 12786 leader_election.cc:334] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c9daca0f91eb486fad1a5699e435f53a (127.5.232.190:35773): Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
I20260929 16:29:35.732224 12787 leader_election.cc:302] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b43f0470b7c49c593a6dc06d8c12162, 6721b56a80e64285978fa1b321a9f761; no voters: c9daca0f91eb486fad1a5699e435f53a
I20260929 16:29:35.732896 13528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:29:35.733016 13528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c9daca0f91eb486fad1a5699e435f53a)
I20260929 16:29:35.733107 13528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:29:35.734934 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.731980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57136)
0929 16:29:35.734352 (+ 2372us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.734564 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2236,"spinlock_wait_cycles":6528,"thread_start_us":2216,"threads_started":1}
I20260929 16:29:35.735270 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.717692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:42019 (local address 127.0.0.1:60626)
0929 16:29:35.717919 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.717948 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:35.718209 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:35.720353 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:35.720361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:35.720405 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:35.720799 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.720815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.723126 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.723135 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.732548 (+ 9413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.732566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.734473 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.734481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.734689 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.734728 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:29:35.734821 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 16:29:35.734988 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 16:29:35.735193 12972 leader_election.cc:334] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c9daca0f91eb486fad1a5699e435f53a (127.5.232.190:35773): Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
I20260929 16:29:35.736284 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.718896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.5.232.189:42019)
0929 16:29:35.719931 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:35.719949 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:35.719979 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:29:35.720050 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:35.720059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:35.720151 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:35.720988 (+ 837us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:35.721248 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.722877 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.732765 (+ 9888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.734208 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.735613 (+ 1405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.735731 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:35.735781 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 16:29:35.735910 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":852,"thread_start_us":638,"threads_started":1}
I20260929 16:29:35.736446 13528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } }
I20260929 16:29:35.736933 12812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b43f0470b7c49c593a6dc06d8c12162" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6721b56a80e64285978fa1b321a9f761" is_pre_election: true
I20260929 16:29:35.737311 12812 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b43f0470b7c49c593a6dc06d8c12162 in current term 2: Already voted for candidate 6721b56a80e64285978fa1b321a9f761 in this term.
I20260929 16:29:35.737355 13528 leader_election.cc:288] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [CANDIDATE]: Term 2 election: Requested vote from peers c9daca0f91eb486fad1a5699e435f53a (127.5.232.190:35773), 0b43f0470b7c49c593a6dc06d8c12162 (127.5.232.188:34743)
I20260929 16:29:35.737968 12999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6721b56a80e64285978fa1b321a9f761" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b43f0470b7c49c593a6dc06d8c12162"
I20260929 16:29:35.738194 12999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:29:35.738554 12973 leader_election.cc:302] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b43f0470b7c49c593a6dc06d8c12162; no voters: 6721b56a80e64285978fa1b321a9f761, c9daca0f91eb486fad1a5699e435f53a
I20260929 16:29:35.738758 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.738072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57150)
0929 16:29:35.738330 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.738530 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:29:35.740175 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.739784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57162)
0929 16:29:35.739956 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.740042 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 16:29:35.740485 12786 leader_election.cc:334] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c9daca0f91eb486fad1a5699e435f53a (127.5.232.190:35773): Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
I20260929 16:29:35.741235 12999 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6721b56a80e64285978fa1b321a9f761 in term 2.
I20260929 16:29:35.741524 13527 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:29:35.741745 12787 leader_election.cc:302] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b43f0470b7c49c593a6dc06d8c12162, 6721b56a80e64285978fa1b321a9f761; no voters: c9daca0f91eb486fad1a5699e435f53a
I20260929 16:29:35.742024 13528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:29:35.742411 13528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [term 2 LEADER]: Becoming Leader. State: Replica: 6721b56a80e64285978fa1b321a9f761, State: Running, Role: LEADER
I20260929 16:29:35.742792 13528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } }
I20260929 16:29:35.745275 13538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6721b56a80e64285978fa1b321a9f761. Latest consensus state: current_term: 2 leader_uuid: "6721b56a80e64285978fa1b321a9f761" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:35.745644 13538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:35.746290 13540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:35.749853 13540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:35.750669 13540 catalog_manager.cc:1259] Loaded cluster ID: fa8e4544e6fb4a42ae8ab152fa0bb3fc
I20260929 16:29:35.750739 13540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:35.752722 13540 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:35.753556 13540 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Loaded TSK: 0
I20260929 16:29:35.754369 13540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:29:35.809685 12802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:35.816892 12802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:35.821312 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.820844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57186)
0929 16:29:35.821030 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.821126 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 16:29:35.821316 12999 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [term 2 FOLLOWER]: Refusing update from remote peer 6721b56a80e64285978fa1b321a9f761: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 16:29:35.822129 13538 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 16:29:35.822371 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.822024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57196)
0929 16:29:35.822157 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.822226 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:29:35.822549 12786 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 -> Peer c9daca0f91eb486fad1a5699e435f53a (127.5.232.190:35773): Couldn't send request to peer c9daca0f91eb486fad1a5699e435f53a. Status: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:29:35.824761 13527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6721b56a80e64285978fa1b321a9f761. Latest consensus state: current_term: 2 leader_uuid: "6721b56a80e64285978fa1b321a9f761" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:35.824978 13527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:29:35.827783 13527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6721b56a80e64285978fa1b321a9f761" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:35.828164 13527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43f0470b7c49c593a6dc06d8c12162 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:29:35.834270 13528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6721b56a80e64285978fa1b321a9f761" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c9daca0f91eb486fad1a5699e435f53a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35773 } } peers { permanent_uuid: "6721b56a80e64285978fa1b321a9f761" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 42019 } attrs { promote: false } } peers { permanent_uuid: "0b43f0470b7c49c593a6dc06d8c12162" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 34743 } attrs { promote: false } } }
I20260929 16:29:35.834621 13528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:35.850107 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.844553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.5.232.129:40865)
0929 16:29:35.845007 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:35.845013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:35.845031 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:29:35.845099 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:35.845105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:35.845171 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:35.845284 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:35.845675 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.846589 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.848275 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.849021 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.849165 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.849272 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:35.849678 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 16:29:35.849806 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":254,"threads_started":1}
I20260929 16:29:35.850080 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.844173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40865 (local address 127.0.0.1:48154)
0929 16:29:35.844349 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.844366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:35.844455 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:35.845294 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:35.845299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:35.845320 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:35.845535 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.845543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.846736 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.846740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.848099 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.848111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.849194 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.849200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.849465 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.849507 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:29:35.849627 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 16:29:35.849787 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":1}
I20260929 16:29:35.851996 13104 tablet_service.cc:1467] Processing CreateTablet for tablet 4953d772aa9a41d995cd17b54e4343a4 (DEFAULT_TABLE table=test-table [id=1060c275012d4d7ca780f8432330aff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:35.852427 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4953d772aa9a41d995cd17b54e4343a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:35.853011 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.842934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:39939 (local address 127.0.0.1:59482)
0929 16:29:35.843106 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.843124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:35.843241 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:35.844109 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:35.844118 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:35.844159 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:35.844596 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.844611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.846752 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.846761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.849598 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.849619 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.851952 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.851961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.852236 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.852293 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 16:29:35.852374 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:29:35.852578 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":1}
I20260929 16:29:35.854372 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.843084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.5.232.131:39939)
0929 16:29:35.843661 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:35.843671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:35.843695 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:29:35.843786 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:35.843794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:35.843890 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:35.844772 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:35.845039 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.846557 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.849839 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.851646 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.853463 (+ 1817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.853639 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:35.853707 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 16:29:35.853905 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":376,"threads_started":1}
I20260929 16:29:35.855305 13392 tablet_service.cc:1467] Processing CreateTablet for tablet 4953d772aa9a41d995cd17b54e4343a4 (DEFAULT_TABLE table=test-table [id=1060c275012d4d7ca780f8432330aff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:35.855875 13392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4953d772aa9a41d995cd17b54e4343a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:35.857707 13550 tablet_bootstrap.cc:492] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: Bootstrap starting.
I20260929 16:29:35.859711 13550 tablet_bootstrap.cc:654] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:35.862087 13550 tablet_bootstrap.cc:492] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: No bootstrap required, opened a new log
I20260929 16:29:35.862524 13550 ts_tablet_manager.cc:1397] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 16:29:35.863453 13550 raft_consensus.cc:348] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } }
I20260929 16:29:35.863664 13550 raft_consensus.cc:374] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:35.863749 13550 raft_consensus.cc:729] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a026605c1054cf3a71540cc71a19432, State: Initialized, Role: FOLLOWER
I20260929 16:29:35.864009 13550 consensus_queue.cc:260] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } }
I20260929 16:29:35.864861 13550 ts_tablet_manager.cc:1428] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:29:35.865881 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.842579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.5.232.130:37611)
0929 16:29:35.857803 (+ 15224us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:35.857812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:35.857833 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:29:35.857903 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:35.857907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:35.857984 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:35.858750 (+ 766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:35.858905 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.859918 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.862091 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.863819 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.864025 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.864180 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:35.865242 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0929 16:29:35.865433 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15091,"thread_start_us":15501,"threads_started":1}
I20260929 16:29:35.866770 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.842440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:37611 (local address 127.0.0.1:58866)
0929 16:29:35.842658 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.842678 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:35.842791 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:35.858239 (+ 15448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:35.858245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:35.858275 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:35.858590 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.858605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.860072 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.860081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.861884 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:35.861901 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:35.864400 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:35.864410 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:35.864711 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:35.864775 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 16:29:35.864890 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:29:35.866320 (+ 1430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 16:29:35.868086 13251 tablet_service.cc:1467] Processing CreateTablet for tablet 4953d772aa9a41d995cd17b54e4343a4 (DEFAULT_TABLE table=test-table [id=1060c275012d4d7ca780f8432330aff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:35.868682 13251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4953d772aa9a41d995cd17b54e4343a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:35.870805 13552 tablet_bootstrap.cc:492] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: Bootstrap starting.
I20260929 16:29:35.873338 13552 tablet_bootstrap.cc:654] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:35.874174 13171 heartbeater.cc:503] Master 127.5.232.189:42019 was elected leader, sending a full tablet report...
I20260929 16:29:35.877938 13556 tablet_bootstrap.cc:492] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: Bootstrap starting.
I20260929 16:29:35.878562 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.865509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.129:51723)
0929 16:29:35.877443 (+ 11934us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.877632 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11738,"thread_start_us":11594,"threads_started":1}
I20260929 16:29:35.879880 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.879524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.129:58649)
0929 16:29:35.879687 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.879777 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:29:35.879912 13556 tablet_bootstrap.cc:654] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:35.880820 13552 tablet_bootstrap.cc:492] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: No bootstrap required, opened a new log
I20260929 16:29:35.880985 13552 ts_tablet_manager.cc:1397] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 16:29:35.881848 13552 raft_consensus.cc:348] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } }
I20260929 16:29:35.882115 13556 tablet_bootstrap.cc:492] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: No bootstrap required, opened a new log
I20260929 16:29:35.882094 13552 raft_consensus.cc:374] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:35.882197 13552 raft_consensus.cc:729] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28cf18f75454f029b1693e7ec5bf493, State: Initialized, Role: FOLLOWER
I20260929 16:29:35.882248 13556 ts_tablet_manager.cc:1397] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:29:35.882453 13552 consensus_queue.cc:260] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } }
I20260929 16:29:35.883039 13556 raft_consensus.cc:348] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } }
I20260929 16:29:35.883262 13552 ts_tablet_manager.cc:1428] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 16:29:35.883272 13556 raft_consensus.cc:374] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:35.883356 13556 raft_consensus.cc:729] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ed68c31236410e9aac71d98835d8df, State: Initialized, Role: FOLLOWER
I20260929 16:29:35.883602 13556 consensus_queue.cc:260] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } }
I20260929 16:29:35.884982 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.884014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.131:57513)
0929 16:29:35.884558 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.884736 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"threads_started":1}
I20260929 16:29:35.885425 13459 heartbeater.cc:503] Master 127.5.232.189:42019 was elected leader, sending a full tablet report...
I20260929 16:29:35.886646 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.886253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.131:40213)
0929 16:29:35.886414 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.886516 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:29:35.888938 13556 ts_tablet_manager.cc:1428] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260929 16:29:35.889536 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.888466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.130:39493)
0929 16:29:35.889064 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.889248 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":286,"threads_started":1}
I20260929 16:29:35.890297 13318 heartbeater.cc:503] Master 127.5.232.189:42019 was elected leader, sending a full tablet report...
I20260929 16:29:35.890683 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:35.890243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.130:36221)
0929 16:29:35.890472 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:35.890550 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260929 16:29:36.037041 13319 tablet.cc:2369] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:29:36.052798 13553 raft_consensus.cc:482] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:36.053035 13553 raft_consensus.cc:504] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } }
I20260929 16:29:36.054677 13553 leader_election.cc:288] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611), c28cf18f75454f029b1693e7ec5bf493 (127.5.232.131:39939)
I20260929 16:29:36.061813 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.055118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:43695 (local address 127.5.232.130:37611)
0929 16:29:36.055344 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:36.055352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:36.055398 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:29:36.055582 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:36.055588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:36.055666 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:36.055763 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:36.056325 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.057424 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.060119 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.060873 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.061051 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:36.061129 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:36.061435 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 16:29:36.061564 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:29:36.061856 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.054918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:37611 (local address 127.5.232.129:43695)
0929 16:29:36.055208 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.055237 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:36.055410 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:36.055813 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:36.055820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:36.055851 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:36.056142 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:36.056177 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.057547 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.057550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:36.059898 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:36.059915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.061051 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.061058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:36.061238 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:36.061270 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:29:36.061386 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 16:29:36.061553 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":35}
I20260929 16:29:36.062000 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.055205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:34441 (local address 127.5.232.131:39939)
0929 16:29:36.055386 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:36.055392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:36.055894 (+ 502us) server_negotiation.cc:407] Connection header received
0929 16:29:36.056028 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:36.056036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:36.056102 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:36.056173 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:36.056654 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.057658 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.060069 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.060958 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.061138 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:36.061216 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:36.061614 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 16:29:36.061742 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:29:36.062165 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.055124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:39939 (local address 127.5.232.129:34441)
0929 16:29:36.055772 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.055789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:36.055904 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:36.056216 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:36.056223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:36.056255 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:36.056519 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:36.056526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.057765 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.057770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:36.059907 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:36.059915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.061178 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.061184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:36.061464 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:36.061489 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:29:36.061589 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:29:36.061712 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":77,"thread_start_us":397,"threads_started":1}
I20260929 16:29:36.062893 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4953d772aa9a41d995cd17b54e4343a4" candidate_uuid: "9a026605c1054cf3a71540cc71a19432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ed68c31236410e9aac71d98835d8df" is_pre_election: true
I20260929 16:29:36.062999 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4953d772aa9a41d995cd17b54e4343a4" candidate_uuid: "9a026605c1054cf3a71540cc71a19432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c28cf18f75454f029b1693e7ec5bf493" is_pre_election: true
I20260929 16:29:36.063174 13271 raft_consensus.cc:2454] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a026605c1054cf3a71540cc71a19432 in term 0.
I20260929 16:29:36.063423 13412 raft_consensus.cc:2454] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a026605c1054cf3a71540cc71a19432 in term 0.
I20260929 16:29:36.063740 13057 leader_election.cc:302] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ed68c31236410e9aac71d98835d8df, 9a026605c1054cf3a71540cc71a19432; no voters:
I20260929 16:29:36.064175 13553 raft_consensus.cc:2781] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:36.064306 13553 raft_consensus.cc:482] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:36.064466 13553 raft_consensus.cc:3037] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:36.066800 13553 raft_consensus.cc:504] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } }
I20260929 16:29:36.067507 13553 leader_election.cc:288] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [CANDIDATE]: Term 1 election: Requested vote from peers 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611), c28cf18f75454f029b1693e7ec5bf493 (127.5.232.131:39939)
I20260929 16:29:36.068145 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4953d772aa9a41d995cd17b54e4343a4" candidate_uuid: "9a026605c1054cf3a71540cc71a19432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c28cf18f75454f029b1693e7ec5bf493"
I20260929 16:29:36.068228 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4953d772aa9a41d995cd17b54e4343a4" candidate_uuid: "9a026605c1054cf3a71540cc71a19432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ed68c31236410e9aac71d98835d8df"
I20260929 16:29:36.068442 13412 raft_consensus.cc:3037] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:36.068449 13271 raft_consensus.cc:3037] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:36.070643 13271 raft_consensus.cc:2454] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a026605c1054cf3a71540cc71a19432 in term 1.
I20260929 16:29:36.070940 13412 raft_consensus.cc:2454] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a026605c1054cf3a71540cc71a19432 in term 1.
I20260929 16:29:36.071099 13057 leader_election.cc:302] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ed68c31236410e9aac71d98835d8df, 9a026605c1054cf3a71540cc71a19432; no voters:
I20260929 16:29:36.071414 13553 raft_consensus.cc:2781] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:36.071866 13553 raft_consensus.cc:686] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 1 LEADER]: Becoming Leader. State: Replica: 9a026605c1054cf3a71540cc71a19432, State: Running, Role: LEADER
I20260929 16:29:36.072297 13553 consensus_queue.cc:237] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } }
I20260929 16:29:36.072885 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.072493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.129:41545)
0929 16:29:36.072652 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.072762 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 16:29:36.076287 12802 catalog_manager.cc:5495] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a026605c1054cf3a71540cc71a19432 (127.5.232.129). New cstate: current_term: 1 leader_uuid: "9a026605c1054cf3a71540cc71a19432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a026605c1054cf3a71540cc71a19432" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40865 } health_report { overall_health: HEALTHY } } }
I20260929 16:29:36.095228 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.090316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.5.232.129:40865)
0929 16:29:36.090510 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:36.090514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:36.090531 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:29:36.090739 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:36.090743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:36.090826 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:36.090972 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:36.091254 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.091956 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.093143 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.093615 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.093708 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:36.094048 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:36.094694 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:36.094798 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:36.094921 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 16:29:36.095015 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:29:36.099675 13271 raft_consensus.cc:1261] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [term 1 FOLLOWER]: Refusing update from remote peer 9a026605c1054cf3a71540cc71a19432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:36.099999 13412 raft_consensus.cc:1261] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [term 1 FOLLOWER]: Refusing update from remote peer 9a026605c1054cf3a71540cc71a19432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:36.100406 13563 consensus_queue.cc:1035] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01ed68c31236410e9aac71d98835d8df" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:36.100821 13553 consensus_queue.cc:1035] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c28cf18f75454f029b1693e7ec5bf493" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:36.103195 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.102683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.131:43669)
0929 16:29:36.102897 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.102987 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 16:29:36.104111 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.103483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.130:35973)
0929 16:29:36.103618 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.103919 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 16:29:36.109072 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.108675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.130:45557)
0929 16:29:36.108835 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.108915 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 16:29:36.110909 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.110509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.129:59861)
0929 16:29:36.110671 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.110757 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:29:36.113317 13567 mvcc.cc:205] Tried to move back new op lower bound from 7334704644496470016 to 7334704644396523520. Current Snapshot: MvccSnapshot[applied={T|T < 7334704644496470016 or (T in {7334704644496470016})}]
I20260929 16:29:36.121047 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.120551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.129:33615)
0929 16:29:36.120745 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.120855 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 16:29:36.121657 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.121167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.5.232.131:60513)
0929 16:29:36.121410 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.121495 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 16:29:36.122020 13568 mvcc.cc:205] Tried to move back new op lower bound from 7334704644496470016 to 7334704644396523520. Current Snapshot: MvccSnapshot[applied={T|T < 7334704644496470016}]
I20260929 16:29:36.126801 12802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:36.127084 12802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:36.132180 12802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6721b56a80e64285978fa1b321a9f761: Sending DeleteTablet for 3 replicas of tablet 4953d772aa9a41d995cd17b54e4343a4
I20260929 16:29:36.133081 13251 tablet_service.cc:1514] Processing DeleteTablet for tablet 4953d772aa9a41d995cd17b54e4343a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:36 UTC) from {username='oryx', principal='oryx/127.5.232.189@KRBTEST.COM'} at 127.0.0.1:58866
I20260929 16:29:36.133796 13392 tablet_service.cc:1514] Processing DeleteTablet for tablet 4953d772aa9a41d995cd17b54e4343a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:36 UTC) from {username='oryx', principal='oryx/127.5.232.189@KRBTEST.COM'} at 127.0.0.1:59482
I20260929 16:29:36.134217 13104 tablet_service.cc:1514] Processing DeleteTablet for tablet 4953d772aa9a41d995cd17b54e4343a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:36 UTC) from {username='oryx', principal='oryx/127.5.232.189@KRBTEST.COM'} at 127.0.0.1:48154
I20260929 16:29:36.134526 13577 tablet_replica.cc:330] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: stopping tablet replica
I20260929 16:29:36.134766 13577 raft_consensus.cc:2229] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:36.135107 13577 raft_consensus.cc:2258] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:36.136394 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 12976
I20260929 16:29:36.136462 13577 ts_tablet_manager.cc:1905] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:36.139796 13578 tablet_replica.cc:330] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: stopping tablet replica
I20260929 16:29:36.139982 13578 raft_consensus.cc:2229] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:29:36.140569 13578 raft_consensus.cc:2258] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:36.141852 13576 tablet_replica.cc:330] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: stopping tablet replica
I20260929 16:29:36.142100 13576 raft_consensus.cc:2229] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:36.142443 13576 raft_consensus.cc:2258] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:36.143640 13576 ts_tablet_manager.cc:1905] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:36.145716 13577 ts_tablet_manager.cc:1918] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:36.145884 13577 log.cc:1196] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/4953d772aa9a41d995cd17b54e4343a4
I20260929 16:29:36.146430 13577 ts_tablet_manager.cc:1939] T 4953d772aa9a41d995cd17b54e4343a4 P c28cf18f75454f029b1693e7ec5bf493: Deleting consensus metadata
I20260929 16:29:36.147893 12788 catalog_manager.cc:4841] TS c28cf18f75454f029b1693e7ec5bf493 (127.5.232.131:39939): tablet 4953d772aa9a41d995cd17b54e4343a4 (table test-table [id=1060c275012d4d7ca780f8432330aff2]) successfully deleted
I20260929 16:29:36.150285 13576 ts_tablet_manager.cc:1918] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:36.150290 13578 ts_tablet_manager.cc:1905] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:36.150417 13576 log.cc:1196] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/4953d772aa9a41d995cd17b54e4343a4
I20260929 16:29:36.150903 13576 ts_tablet_manager.cc:1939] T 4953d772aa9a41d995cd17b54e4343a4 P 01ed68c31236410e9aac71d98835d8df: Deleting consensus metadata
I20260929 16:29:36.151970 12786 catalog_manager.cc:4841] TS 01ed68c31236410e9aac71d98835d8df (127.5.232.130:37611): tablet 4953d772aa9a41d995cd17b54e4343a4 (table test-table [id=1060c275012d4d7ca780f8432330aff2]) successfully deleted
I20260929 16:29:36.156832 13578 ts_tablet_manager.cc:1918] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:36.156988 13578 log.cc:1196] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/4953d772aa9a41d995cd17b54e4343a4
I20260929 16:29:36.157482 13578 ts_tablet_manager.cc:1939] T 4953d772aa9a41d995cd17b54e4343a4 P 9a026605c1054cf3a71540cc71a19432: Deleting consensus metadata
I20260929 16:29:36.158444 12786 catalog_manager.cc:4841] TS 9a026605c1054cf3a71540cc71a19432 (127.5.232.129:40865): tablet 4953d772aa9a41d995cd17b54e4343a4 (table test-table [id=1060c275012d4d7ca780f8432330aff2]) successfully deleted
I20260929 16:29:36.170290 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.137453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.5.232.129:40865)
0929 16:29:36.137621 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:36.137628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:36.149721 (+ 12093us) server_negotiation.cc:407] Connection header received
0929 16:29:36.150175 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:36.150181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:36.150262 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:36.150350 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:36.152310 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.153603 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.155357 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.156082 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.156229 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:36.163000 (+ 6771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:36.163018 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:36.163039 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:36.163096 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:36.167111 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:36.167892 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:36.167905 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:36.167909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:36.168133 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:36.168554 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:36.168560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:36.168563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:36.169030 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:36.169364 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:36.169654 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 16:29:36.169894 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:29:36.171010 13164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:36.275055 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 12976
I20260929 16:29:36.293577 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 13181
I20260929 16:29:36.306819 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.294518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.5.232.130:37611)
0929 16:29:36.294707 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:36.294713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:36.294734 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:29:36.294791 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:36.294797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:36.294902 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:36.295001 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:36.296454 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.297268 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.298601 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.298995 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.299096 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:36.301453 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:36.301468 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:36.301485 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:36.301523 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:36.304348 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:36.304958 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:36.304966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:36.304968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:36.305087 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:36.305465 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:36.305468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:36.305470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:36.305814 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:36.306041 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:36.306289 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 16:29:36.306452 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:29:36.307503 13311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:36.403452 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 13181
I20260929 16:29:36.408684 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.407996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57208)
0929 16:29:36.408451 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.408533 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":251,"threads_started":1}
I20260929 16:29:36.410095 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.409729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35773 (local address 127.0.0.1:57212)
0929 16:29:36.409890 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:36.409970 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:29:36.421463 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 13329
I20260929 16:29:36.434926 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:36.422198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.5.232.131:39939)
0929 16:29:36.422390 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:36.422395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:36.422415 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:29:36.422604 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:36.422610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:36.422693 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:36.422799 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:36.424104 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.424871 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.426362 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:36.426774 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:36.426915 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:36.429370 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:36.429427 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:36.429456 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:36.429502 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:36.432130 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:36.432796 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:36.432806 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:36.432810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:36.432956 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:36.433325 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:36.433331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:36.433335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:36.433753 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:36.434057 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:36.434362 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 16:29:36.434586 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:29:36.435586 13452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:36.556383 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 13329
I20260929 16:29:36.589362 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 12665
I20260929 16:29:36.590747 12832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:36.714917 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 12665
W20260929 16:29:36.735886 12973 connection.cc:535] server connection from 127.0.0.1:48124 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:29:36.736344 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 12766
I20260929 16:29:36.737360 13019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:36.851250 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 12766
2026-09-29T16:29:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9622 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:36 dist-test-slave-dssx krb5kdc[13589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:36 dist-test-slave-dssx krb5kdc[13589](info): set up 2 sockets
Sep 29 16:29:36 dist-test-slave-dssx krb5kdc[13589](info): commencing operation
krb5kdc: starting...
W20260929 16:29:37.942346 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:29:38 dist-test-slave-dssx krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.190.keytab.
Entry for principal oryx/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.190.keytab.
I20260929 16:29:38.161132 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:44087
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.190.keytab
--principal=oryx/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:44087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:38.496645 13602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:38.496956 13602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:38.497121 13602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:38.504575 13602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:38.504694 13602 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:29:38.504734 13602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:38.504861 13602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:38.504953 13602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:38.505070 13602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:38.515251 13602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:44087
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.190.keytab
--principal=oryx/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:44087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:38.516654 13602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:38.518131 13602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:38.530933 13607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:38.530833 13610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:38.530880 13608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:38.531703 13602 server_base.cc:1034] running on GCE node
I20260929 16:29:38.532570 13602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:38.534162 13602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:38.535411 13602 hybrid_clock.cc:648] HybridClock initialized: now 1790699378535333 us; error 47 us; skew 500 ppm
I20260929 16:29:38.535749 13602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:38 dist-test-slave-dssx krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699378, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:38.539050 13602 init.cc:377] Logged in from keytab as oryx/127.5.232.190@KRBTEST.COM (short username oryx)
I20260929 16:29:38.540616 13602 webserver.cc:460] Webserver started at http://127.5.232.190:38085/ using document root <none> and password file <none>
I20260929 16:29:38.541471 13602 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:38.541604 13602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:38.541960 13602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:38.545127 13602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "6516f04fa06244f4aae5d7ae404035b8"
format_stamp: "Formatted at 2026-09-29 16:29:38 on dist-test-slave-dssx"
server_key: "c70b91eb860fa48bd0df4866cd547352"
server_key_iv: "74f65149e2738d22db57aedfbd50e15a"
server_key_version: "encryptionkey@0"
I20260929 16:29:38.546075 13602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "6516f04fa06244f4aae5d7ae404035b8"
format_stamp: "Formatted at 2026-09-29 16:29:38 on dist-test-slave-dssx"
server_key: "c70b91eb860fa48bd0df4866cd547352"
server_key_iv: "74f65149e2738d22db57aedfbd50e15a"
server_key_version: "encryptionkey@0"
I20260929 16:29:38.551589 13602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:29:38.555384 13617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:38.556845 13602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:38.557046 13602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "6516f04fa06244f4aae5d7ae404035b8"
format_stamp: "Formatted at 2026-09-29 16:29:38 on dist-test-slave-dssx"
server_key: "c70b91eb860fa48bd0df4866cd547352"
server_key_iv: "74f65149e2738d22db57aedfbd50e15a"
server_key_version: "encryptionkey@0"
I20260929 16:29:38.557287 13602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:38.587855 13602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:38.592209 13602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:38.592520 13602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:38.615063 13602 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:44087
I20260929 16:29:38.615070 13678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:44087 every 8 connection(s)
I20260929 16:29:38.616576 13602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:38.620822 13679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:38.623884 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 13602
I20260929 16:29:38.624130 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:38.624950 6050 external_mini_cluster.cc:1428] Setting key ed21bbc1ac258ea1faf5624ce77e5978
I20260929 16:29:38.634385 13679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8: Bootstrap starting.
I20260929 16:29:38.638257 13679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:38.639657 13679 log.cc:824] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:29:38 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:38.642901 13679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8: No bootstrap required, opened a new log
I20260929 16:29:38.648991 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:38.627271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.5.232.190:44087)
0929 16:29:38.628101 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:38.628114 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:38.628166 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:29:38.629956 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:38.629981 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:38.630432 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:38.630877 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:38.632668 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:38.634018 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:38.635726 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:38.636428 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:38.636566 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:38.641431 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:38.641447 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:38.641463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:38.641499 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:38.644551 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:38.645297 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:38.645312 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:38.645319 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:38.645565 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:38.645984 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:38.645989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:38.645993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:38.646581 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:38.646902 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:38.647373 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 16:29:38.647828 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":388,"threads_started":1}
I20260929 16:29:38.651027 13679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6516f04fa06244f4aae5d7ae404035b8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44087 } }
I20260929 16:29:38.651425 13679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:38.651563 13679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6516f04fa06244f4aae5d7ae404035b8, State: Initialized, Role: FOLLOWER
I20260929 16:29:38.652218 13679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6516f04fa06244f4aae5d7ae404035b8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44087 } }
I20260929 16:29:38.652452 13679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:38.652583 13679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:38.652750 13679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:38.655159 13679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6516f04fa06244f4aae5d7ae404035b8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44087 } }
I20260929 16:29:38.655747 13679 leader_election.cc:302] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6516f04fa06244f4aae5d7ae404035b8; no voters:
I20260929 16:29:38.656342 13679 leader_election.cc:288] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:38.656481 13684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:38.657207 13684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6516f04fa06244f4aae5d7ae404035b8, State: Running, Role: LEADER
I20260929 16:29:38.657817 13684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6516f04fa06244f4aae5d7ae404035b8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44087 } }
I20260929 16:29:38.658701 13679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:38.660750 13685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6516f04fa06244f4aae5d7ae404035b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6516f04fa06244f4aae5d7ae404035b8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44087 } } }
I20260929 16:29:38.661173 13685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:38.662431 13686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6516f04fa06244f4aae5d7ae404035b8. Latest consensus state: current_term: 1 leader_uuid: "6516f04fa06244f4aae5d7ae404035b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6516f04fa06244f4aae5d7ae404035b8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44087 } } }
I20260929 16:29:38.662793 13686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:38.667047 13689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:38.673302 13689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:38.683223 13689 catalog_manager.cc:1347] Generated new cluster ID: 2436d3ecc35543e194a0f8d3ccedb2d9
I20260929 16:29:38.683355 13689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:38.712185 13689 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:38.713708 13689 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:38.724050 13689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8: Generated new TSK 0
I20260929 16:29:38.725216 13689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab.
Entry for principal oryx/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab.
I20260929 16:29:38.781271 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44087
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab
--principal=oryx/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:39.127276 13705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:39.127635 13705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:39.127812 13705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:39.135406 13705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:39.135581 13705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:39.135735 13705 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:29:39.135874 13705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:39.146057 13705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab
--principal=oryx/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.232.190:44087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:39.147441 13705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:39.148901 13705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:39.163379 13713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:39.163609 13711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:39.163751 13710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:39.169771 13705 server_base.cc:1034] running on GCE node
I20260929 16:29:39.170456 13705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:39.171427 13705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:39.172672 13705 hybrid_clock.cc:648] HybridClock initialized: now 1790699379172581 us; error 66 us; skew 500 ppm
I20260929 16:29:39.172997 13705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:39 dist-test-slave-dssx krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699379, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:39.176390 13705 init.cc:377] Logged in from keytab as oryx/127.5.232.129@KRBTEST.COM (short username oryx)
I20260929 16:29:39.177982 13705 webserver.cc:460] Webserver started at http://127.5.232.129:36331/ using document root <none> and password file <none>
I20260929 16:29:39.178740 13705 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:39.178834 13705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:39.179121 13705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:39.182225 13705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "bad05bea81614f598e72f9e066d7b460"
format_stamp: "Formatted at 2026-09-29 16:29:39 on dist-test-slave-dssx"
server_key: "daaef2439dab300627fe1bccaf3d1dbd"
server_key_iv: "4e1a259ff2188659100b37c10aae0b91"
server_key_version: "encryptionkey@0"
I20260929 16:29:39.183027 13705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "bad05bea81614f598e72f9e066d7b460"
format_stamp: "Formatted at 2026-09-29 16:29:39 on dist-test-slave-dssx"
server_key: "daaef2439dab300627fe1bccaf3d1dbd"
server_key_iv: "4e1a259ff2188659100b37c10aae0b91"
server_key_version: "encryptionkey@0"
I20260929 16:29:39.188475 13705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:29:39.192065 13720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:39.193572 13705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:39.193717 13705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "bad05bea81614f598e72f9e066d7b460"
format_stamp: "Formatted at 2026-09-29 16:29:39 on dist-test-slave-dssx"
server_key: "daaef2439dab300627fe1bccaf3d1dbd"
server_key_iv: "4e1a259ff2188659100b37c10aae0b91"
server_key_version: "encryptionkey@0"
I20260929 16:29:39.193912 13705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:39.227367 13705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:39.231539 13705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:39.231874 13705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:39.234792 13705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:39.234899 13705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:39.235065 13705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:39.235152 13705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:29:39 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699379, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:39.262115 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:39.235330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:43343 (local address 127.5.232.190:44087)
0929 16:29:39.236166 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:39.236171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:39.237229 (+ 1058us) server_negotiation.cc:407] Connection header received
0929 16:29:39.239446 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:39.239451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:39.239545 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:39.239679 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:39.242235 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.243606 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.246734 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.247375 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.247545 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:39.253715 (+ 6170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:39.253726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:39.253732 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:39.253784 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:39.257493 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:39.258718 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:39.258730 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:39.258732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:39.258908 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:39.259578 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:39.259585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:39.259589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:39.260019 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:39.260203 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:39.261488 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0929 16:29:39.261727 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":586,"threads_started":1}
I20260929 16:29:39.263238 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:39.236143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44087 (local address 127.5.232.129:43343)
0929 16:29:39.237044 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:39.237092 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:39.239129 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:39.239960 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:39.239992 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:39.240925 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:39.241941 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:39.241977 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.243844 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.243852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:39.246532 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:39.246551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.247736 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.247747 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:39.248192 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:39.249325 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:39.249369 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:39.253292 (+ 3923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:39.257758 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:39.257774 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:39.257797 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:39.258563 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:39.259042 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:39.259047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:39.259051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:39.259389 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:39.260205 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:39.260218 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:39.260974 (+ 756us) client_negotiation.cc:769] Sending connection context
0929 16:29:39.261340 (+ 366us) client_negotiation.cc:240] Negotiation successful
0929 16:29:39.261961 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":408,"threads_started":1}
I20260929 16:29:39.278753 13705 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:44959
I20260929 16:29:39.278766 13840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:44959 every 8 connection(s)
I20260929 16:29:39.280316 13705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:39.282801 13841 heartbeater.cc:347] Connected to a master server at 127.5.232.190:44087
I20260929 16:29:39.283087 13841 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:39.283779 13841 heartbeater.cc:511] Master 127.5.232.190:44087 requested a full tablet report, sending...
I20260929 16:29:39.286273 13634 ts_manager.cc:194] Registered new tserver with Master: bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959)
I20260929 16:29:39.286623 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:39.271191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:54211 (local address 127.5.232.190:44087)
0929 16:29:39.271453 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:39.271462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:39.272798 (+ 1336us) server_negotiation.cc:407] Connection header received
0929 16:29:39.273026 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:39.273034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:39.273133 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:39.273240 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:39.274895 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.275981 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.277892 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.278336 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.278486 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:39.280558 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:39.280564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:39.280567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:39.280599 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:39.283623 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:39.284415 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:39.284421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:39.284425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:39.284572 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:39.284986 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:39.284991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:39.284994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:39.285411 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:39.285585 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:39.285962 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 16:29:39.286194 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:29:39.286628 13825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:39.271921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44087 (local address 127.5.232.129:54211)
0929 16:29:39.272669 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:39.272687 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:39.272889 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:39.273481 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:39.273484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:39.274158 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:39.274701 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:39.274721 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.276137 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.276142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:39.277688 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:39.277709 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.278558 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.278565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:39.278764 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:39.279523 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:39.279541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:39.280298 (+ 757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:39.283761 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:39.283771 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:39.283775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:39.284270 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:39.284692 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:39.284697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:39.284700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:39.284834 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:39.285550 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:39.285554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:39.285706 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 16:29:39.285828 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:29:39.286124 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":475,"threads_started":1}
I20260929 16:29:39.286617 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 13705
I20260929 16:29:39.286823 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:39.287585 6050 external_mini_cluster.cc:1428] Setting key f084d869b7811a2c0dd431e685173797
I20260929 16:29:39.288488 13634 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:43343
WARNING: no policy specified for oryx/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab.
Entry for principal oryx/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab.
I20260929 16:29:39.336366 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44087
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab
--principal=oryx/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:39.673087 13845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:39.673511 13845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:39.673683 13845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:39.681564 13845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:39.681730 13845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:39.681888 13845 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:29:39.682008 13845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:39.692193 13845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab
--principal=oryx/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.232.190:44087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:39.693611 13845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:39.694984 13845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:39.708516 13850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:39.708606 13853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:39.708555 13851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:39.709443 13845 server_base.cc:1034] running on GCE node
I20260929 16:29:39.710101 13845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:39.711035 13845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:39.712268 13845 hybrid_clock.cc:648] HybridClock initialized: now 1790699379712198 us; error 48 us; skew 500 ppm
I20260929 16:29:39.712600 13845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:39 dist-test-slave-dssx krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699379, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:39.715858 13845 init.cc:377] Logged in from keytab as oryx/127.5.232.130@KRBTEST.COM (short username oryx)
I20260929 16:29:39.717571 13845 webserver.cc:460] Webserver started at http://127.5.232.130:41371/ using document root <none> and password file <none>
I20260929 16:29:39.718333 13845 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:39.718425 13845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:39.718797 13845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:39.722931 13845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "4c703ebf25d24a3ebefd7163855efa42"
format_stamp: "Formatted at 2026-09-29 16:29:39 on dist-test-slave-dssx"
server_key: "7c5d9bd87ce7e42597904c8a16ce6f5b"
server_key_iv: "426745503c0db17e6e1d4b75dda97cff"
server_key_version: "encryptionkey@0"
I20260929 16:29:39.724169 13845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "4c703ebf25d24a3ebefd7163855efa42"
format_stamp: "Formatted at 2026-09-29 16:29:39 on dist-test-slave-dssx"
server_key: "7c5d9bd87ce7e42597904c8a16ce6f5b"
server_key_iv: "426745503c0db17e6e1d4b75dda97cff"
server_key_version: "encryptionkey@0"
I20260929 16:29:39.731037 13845 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 16:29:39.734917 13860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:39.736449 13845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:39.736654 13845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "4c703ebf25d24a3ebefd7163855efa42"
format_stamp: "Formatted at 2026-09-29 16:29:39 on dist-test-slave-dssx"
server_key: "7c5d9bd87ce7e42597904c8a16ce6f5b"
server_key_iv: "426745503c0db17e6e1d4b75dda97cff"
server_key_version: "encryptionkey@0"
I20260929 16:29:39.736904 13845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:39.752166 13845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:39.756745 13845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:39.757040 13845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:39.760000 13845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:39.760136 13845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:39.760313 13845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:39.760373 13845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:29:39 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699379, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:39.789953 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:39.760884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:37807 (local address 127.5.232.190:44087)
0929 16:29:39.761063 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:39.761069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:39.762794 (+ 1725us) server_negotiation.cc:407] Connection header received
0929 16:29:39.766138 (+ 3344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:39.766147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:39.766253 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:39.766369 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:39.768807 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.769978 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.773688 (+ 3710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.774460 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.774615 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:39.780759 (+ 6144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:39.780769 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:39.780776 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:39.780826 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:39.784856 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:39.786321 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:39.786329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:39.786333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:39.786541 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:39.787275 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:39.787283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:39.787287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:39.787741 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:39.787927 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:39.789207 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0929 16:29:39.789525 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:29:39.791859 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:39.761619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44087 (local address 127.5.232.130:37807)
0929 16:29:39.762599 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:39.762653 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:39.765715 (+ 3062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:39.766746 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:39.766759 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:39.767538 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:39.768545 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:39.768586 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.770177 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.770187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:39.773477 (+ 3290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:39.773500 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.774787 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.774797 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:39.775207 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:39.776208 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:39.776250 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:39.780400 (+ 4150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:39.785062 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:39.785082 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:39.785111 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:39.786071 (+ 960us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:39.786685 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:39.786693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:39.786699 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:39.787104 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:39.787918 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:39.787936 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:39.788606 (+ 670us) client_negotiation.cc:769] Sending connection context
0929 16:29:39.789052 (+ 446us) client_negotiation.cc:240] Negotiation successful
0929 16:29:39.789891 (+ 839us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":418,"threads_started":1}
I20260929 16:29:39.801429 13845 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:43763
I20260929 16:29:39.801468 13979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:43763 every 8 connection(s)
I20260929 16:29:39.802706 13845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:39.804970 13980 heartbeater.cc:347] Connected to a master server at 127.5.232.190:44087
I20260929 16:29:39.805200 13980 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:39.806098 13980 heartbeater.cc:511] Master 127.5.232.190:44087 requested a full tablet report, sending...
I20260929 16:29:39.807675 13633 ts_manager.cc:194] Registered new tserver with Master: 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130:43763)
I20260929 16:29:39.808203 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 13845
I20260929 16:29:39.808396 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:39.808895 13633 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:37807
I20260929 16:29:39.809005 6050 external_mini_cluster.cc:1428] Setting key 5677b1f256cdce0fbdba66a03ce44571
I20260929 16:29:39.814280 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:39.799658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:59985 (local address 127.5.232.190:44087)
0929 16:29:39.799870 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:39.799879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:39.800635 (+ 756us) server_negotiation.cc:407] Connection header received
0929 16:29:39.800869 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:39.800874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:39.800965 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:39.801089 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:39.802216 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.802998 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.804821 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.805505 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.805651 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:39.807543 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:39.807548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:39.807551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:39.807582 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:39.811421 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:39.812281 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:39.812286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:39.812290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:39.812426 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:39.812798 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:39.812804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:39.812808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:39.813129 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:39.813285 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:39.813713 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 16:29:39.813935 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:29:39.814379 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:39.799712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44087 (local address 127.5.232.130:59985)
0929 16:29:39.800491 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:39.800519 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:39.800710 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:39.801112 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:39.801117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:39.801773 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:39.802067 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:39.802078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.803132 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.803140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:39.804677 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:39.804689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:39.805684 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:39.805690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:39.805833 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:39.806478 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:39.806494 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:39.807389 (+ 895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:39.811588 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:39.811593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:39.811598 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:39.812150 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:39.812542 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:39.812548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:39.812552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:39.812685 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:39.813248 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:39.813253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:39.813469 (+ 216us) client_negotiation.cc:769] Sending connection context
0929 16:29:39.813627 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 16:29:39.813935 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":372,"threads_started":1}
WARNING: no policy specified for oryx/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab.
Entry for principal oryx/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab.
I20260929 16:29:39.854633 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:44087
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab
--principal=oryx/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:40.185097 13984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:40.185431 13984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:40.185559 13984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:40.192787 13984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:40.192905 13984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:40.192988 13984 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:29:40.193077 13984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:40.203487 13984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab
--principal=oryx/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.232.190:44087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:40.204829 13984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:40.206310 13984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:40.219456 13992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:40.219435 13989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:40.219374 13990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:40.220287 13984 server_base.cc:1034] running on GCE node
I20260929 16:29:40.220917 13984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:40.221956 13984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:40.223207 13984 hybrid_clock.cc:648] HybridClock initialized: now 1790699380223130 us; error 57 us; skew 500 ppm
I20260929 16:29:40.223507 13984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:40 dist-test-slave-dssx krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699380, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:40.226877 13984 init.cc:377] Logged in from keytab as oryx/127.5.232.131@KRBTEST.COM (short username oryx)
I20260929 16:29:40.228345 13984 webserver.cc:460] Webserver started at http://127.5.232.131:38049/ using document root <none> and password file <none>
I20260929 16:29:40.228994 13984 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:40.229089 13984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:40.229351 13984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:40.232323 13984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "e60a69e849ba4d708c87fa074398bd95"
format_stamp: "Formatted at 2026-09-29 16:29:40 on dist-test-slave-dssx"
server_key: "71a02f6f0ecaf6ea15c3dbfc7af35a41"
server_key_iv: "5b47a611d0308b95f02fddfe4e79094d"
server_key_version: "encryptionkey@0"
I20260929 16:29:40.233107 13984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "e60a69e849ba4d708c87fa074398bd95"
format_stamp: "Formatted at 2026-09-29 16:29:40 on dist-test-slave-dssx"
server_key: "71a02f6f0ecaf6ea15c3dbfc7af35a41"
server_key_iv: "5b47a611d0308b95f02fddfe4e79094d"
server_key_version: "encryptionkey@0"
I20260929 16:29:40.238517 13984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:29:40.242019 13999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:40.243353 13984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:29:40.243530 13984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "e60a69e849ba4d708c87fa074398bd95"
format_stamp: "Formatted at 2026-09-29 16:29:40 on dist-test-slave-dssx"
server_key: "71a02f6f0ecaf6ea15c3dbfc7af35a41"
server_key_iv: "5b47a611d0308b95f02fddfe4e79094d"
server_key_version: "encryptionkey@0"
I20260929 16:29:40.243752 13984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:40.266753 13984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:40.271499 13984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:40.271802 13984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:40.274748 13984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:40.274857 13984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:40.275003 13984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:40.275085 13984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:29:40 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699380, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:40.291951 13841 heartbeater.cc:503] Master 127.5.232.190:44087 was elected leader, sending a full tablet report...
I20260929 16:29:40.298772 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:40.275211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:58667 (local address 127.5.232.190:44087)
0929 16:29:40.275406 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:40.275413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:40.277015 (+ 1602us) server_negotiation.cc:407] Connection header received
0929 16:29:40.279318 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:40.279323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:40.279392 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:40.279449 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:40.281905 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.283161 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.284850 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.285625 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.285755 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:40.290537 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:40.290543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:40.290546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:40.290580 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:40.294322 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:40.295418 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:40.295425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:40.295429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:40.295620 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:40.296365 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:40.296371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:40.296374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:40.296773 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:40.296905 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:40.298242 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0929 16:29:40.298420 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:29:40.300680 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:40.275941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44087 (local address 127.5.232.131:58667)
0929 16:29:40.276831 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:40.276871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:40.278920 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:40.280040 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:40.280051 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:40.280916 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:40.281692 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:40.281727 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.283352 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.283355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:40.284693 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:40.284704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.285856 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.285863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:40.286218 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:40.287121 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:40.287149 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:40.290292 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:40.294499 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:40.294511 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:40.294538 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:40.295250 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:40.295810 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:40.295818 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:40.295824 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:40.296182 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:40.296909 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:40.296922 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:40.297594 (+ 672us) client_negotiation.cc:769] Sending connection context
0929 16:29:40.298001 (+ 407us) client_negotiation.cc:240] Negotiation successful
0929 16:29:40.298750 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":410,"threads_started":1}
I20260929 16:29:40.325011 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:40.307288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:52343 (local address 127.5.232.190:44087)
0929 16:29:40.307522 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:40.307546 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:40.308131 (+ 585us) server_negotiation.cc:407] Connection header received
0929 16:29:40.308352 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:40.308357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:40.308451 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:40.308617 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:40.310497 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.311507 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.313484 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.314296 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.314461 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:40.317758 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:40.317764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:40.317768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:40.317805 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:40.321350 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:40.322507 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:40.322516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:40.322522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:40.322712 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:40.323255 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:40.323263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:40.323269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:40.323693 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:40.323877 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:40.324386 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 16:29:40.324631 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:29:40.325142 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:40.307399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44087 (local address 127.5.232.131:52343)
0929 16:29:40.308015 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:40.308035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:40.308210 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:40.308863 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:40.308870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:40.309721 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:40.310302 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:40.310324 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.311655 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.311660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:40.313261 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:40.313286 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.314774 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.314783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:40.315037 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:40.315830 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:40.315860 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:40.317460 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:40.321518 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:40.321523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:40.321563 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:40.322309 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:40.322880 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:40.322891 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:40.322896 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:40.323090 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:40.323867 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:40.323876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:40.324110 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 16:29:40.324271 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:29:40.324655 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":345,"threads_started":1}
I20260929 16:29:40.330296 13984 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:34027
I20260929 16:29:40.330324 14118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:34027 every 8 connection(s)
I20260929 16:29:40.331604 13984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:40.333595 14119 heartbeater.cc:347] Connected to a master server at 127.5.232.190:44087
I20260929 16:29:40.333828 14119 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:40.334578 14119 heartbeater.cc:511] Master 127.5.232.190:44087 requested a full tablet report, sending...
I20260929 16:29:40.335867 13633 ts_manager.cc:194] Registered new tserver with Master: e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027)
I20260929 16:29:40.336740 13633 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:58667
I20260929 16:29:40.337719 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 13984
I20260929 16:29:40.337918 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:40.338451 6050 external_mini_cluster.cc:1428] Setting key 5b8a054524e0dcc03fe9f1d650d9706b
I20260929 16:29:40.341920 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:29:40 dist-test-slave-dssx krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699380, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:29:40 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:40 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:40.812460 13980 heartbeater.cc:503] Master 127.5.232.190:44087 was elected leader, sending a full tablet report...
I20260929 16:29:41.340283 14119 heartbeater.cc:503] Master 127.5.232.190:44087 was elected leader, sending a full tablet report...
W20260929 16:29:41.446554 14127 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.060s user 0.002s sys 0.012s
W20260929 16:29:41.447855 13729 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:40.381142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.5.232.190:44087)
0929 16:29:40.381308 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:40.381312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:40.381751 (+ 439us) server_negotiation.cc:407] Connection header received
0929 16:29:40.381904 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:40.381909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:40.381976 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:40.382066 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:40.383375 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.384116 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.385435 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:40.385853 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:40.385954 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.447561 (+1061607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60054: BlockingRecv error: recv got EOF from 127.0.0.1:60054 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 16:29:41 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699380, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:41.470335 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.451604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.5.232.190:44087)
0929 16:29:41.451859 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:41.451868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:41.452164 (+ 296us) server_negotiation.cc:407] Connection header received
0929 16:29:41.452312 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:41.452316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:41.452399 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:41.452487 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:41.453786 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.454520 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.455777 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.456219 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.456320 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.464788 (+ 8468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:41.464794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:41.464798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:41.464839 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:41.467657 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:41.468455 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:41.468458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:41.468461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:41.468587 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:41.468922 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:41.468925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:41.468927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:41.469215 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:41.469615 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:41.469810 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 16:29:41.469985 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:29:41.474310 13633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:41.482074 13633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:41.509543 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.494990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:34027 (local address 127.0.0.1:49768)
0929 16:29:41.496063 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:41.496105 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:41.496279 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:41.497419 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:41.497430 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:41.497473 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:41.498032 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.498049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.500826 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.500835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.504305 (+ 3470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.504318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.508377 (+ 4059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.508383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.508646 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.508701 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 16:29:41.508830 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:29:41.509051 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":3,"thread_start_us":6425,"threads_started":1}
I20260929 16:29:41.510568 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.495108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.5.232.131:34027)
0929 16:29:41.496323 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:41.496336 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:41.496376 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:29:41.496495 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:41.496507 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:41.496841 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:41.497078 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:41.498321 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.500600 (+ 2279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.504467 (+ 3867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.507838 (+ 3371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.507989 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.508230 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:41.510080 (+ 1850us) server_negotiation.cc:299] Negotiation successful
0929 16:29:41.510287 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":905,"threads_started":1}
I20260929 16:29:41.512281 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.495420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:43763 (local address 127.0.0.1:39294)
0929 16:29:41.501693 (+ 6273us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:41.501721 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:41.501859 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:41.502845 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:41.502850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:41.502872 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:41.504920 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.504931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.506836 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.506841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.509028 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.509043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.510999 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.511005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.511176 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.511224 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 16:29:41.511727 (+ 503us) client_negotiation.cc:240] Negotiation successful
0929 16:29:41.511880 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6132,"mutex_wait_us":1559,"thread_start_us":689,"threads_started":1}
I20260929 16:29:41.513011 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.495943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.5.232.130:43763)
0929 16:29:41.501294 (+ 5351us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:41.501307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:41.501999 (+ 692us) server_negotiation.cc:407] Connection header received
0929 16:29:41.502115 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:41.502125 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:41.502407 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:41.502612 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:41.505111 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.506642 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.509213 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.510762 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.510939 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.511159 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:41.512516 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0929 16:29:41.512697 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5139,"thread_start_us":4969,"threads_started":1}
I20260929 16:29:41.516266 14049 tablet_service.cc:1467] Processing CreateTablet for tablet 195360cc753b449bb763eb9446a64cc2 (DEFAULT_TABLE table=test-table [id=16408012499d4031b1d95419787bc13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:41.516571 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 195360cc753b449bb763eb9446a64cc2 (DEFAULT_TABLE table=test-table [id=16408012499d4031b1d95419787bc13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:41.518076 14049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195360cc753b449bb763eb9446a64cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:41.518076 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195360cc753b449bb763eb9446a64cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:41.520220 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.502144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.5.232.129:44959)
0929 16:29:41.514052 (+ 11908us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:41.514059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:41.514089 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:29:41.514167 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:41.514173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:41.514367 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:41.514584 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:41.514943 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.515900 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.517775 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.519187 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.519356 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.519582 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:41.519769 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 16:29:41.519929 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10820,"thread_start_us":10682,"threads_started":1}
I20260929 16:29:41.520216 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.501863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:44959 (local address 127.0.0.1:47832)
0929 16:29:41.509788 (+ 7925us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:41.509808 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:41.509909 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:41.514562 (+ 4653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:41.514567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:41.514598 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:41.514789 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.514810 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.516052 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.516059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.517559 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.517578 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.519368 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.519374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.519560 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.519596 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:29:41.519723 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 16:29:41.519878 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7819,"thread_start_us":9457,"threads_started":1}
I20260929 16:29:41.523722 13771 tablet_service.cc:1467] Processing CreateTablet for tablet 195360cc753b449bb763eb9446a64cc2 (DEFAULT_TABLE table=test-table [id=16408012499d4031b1d95419787bc13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:41.525110 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195360cc753b449bb763eb9446a64cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:41.535888 14144 tablet_bootstrap.cc:492] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: Bootstrap starting.
I20260929 16:29:41.538283 14146 tablet_bootstrap.cc:492] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: Bootstrap starting.
I20260929 16:29:41.539476 14145 tablet_bootstrap.cc:492] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: Bootstrap starting.
I20260929 16:29:41.540091 14144 tablet_bootstrap.cc:654] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:41.541497 14144 log.cc:824] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:41.542086 14146 tablet_bootstrap.cc:654] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:41.542955 14145 tablet_bootstrap.cc:654] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:41.543368 14146 log.cc:824] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:41.544634 14145 log.cc:824] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:41.544821 14144 tablet_bootstrap.cc:492] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: No bootstrap required, opened a new log
I20260929 16:29:41.545100 14144 ts_tablet_manager.cc:1397] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 16:29:41.547494 14146 tablet_bootstrap.cc:492] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: No bootstrap required, opened a new log
I20260929 16:29:41.547801 14146 ts_tablet_manager.cc:1397] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260929 16:29:41.548678 14145 tablet_bootstrap.cc:492] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: No bootstrap required, opened a new log
I20260929 16:29:41.548938 14145 ts_tablet_manager.cc:1397] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260929 16:29:41.553334 14144 raft_consensus.cc:348] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } }
I20260929 16:29:41.553822 14144 raft_consensus.cc:374] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:41.553920 14144 raft_consensus.cc:729] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c703ebf25d24a3ebefd7163855efa42, State: Initialized, Role: FOLLOWER
I20260929 16:29:41.554564 14144 consensus_queue.cc:260] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } }
I20260929 16:29:41.557893 14144 ts_tablet_manager.cc:1428] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260929 16:29:41.559221 14146 raft_consensus.cc:348] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } }
I20260929 16:29:41.559747 14146 raft_consensus.cc:374] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:41.559856 14146 raft_consensus.cc:729] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad05bea81614f598e72f9e066d7b460, State: Initialized, Role: FOLLOWER
I20260929 16:29:41.559952 14145 raft_consensus.cc:348] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } }
I20260929 16:29:41.560359 14145 raft_consensus.cc:374] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:41.560459 14145 raft_consensus.cc:729] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60a69e849ba4d708c87fa074398bd95, State: Initialized, Role: FOLLOWER
I20260929 16:29:41.560539 14146 consensus_queue.cc:260] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } }
I20260929 16:29:41.561030 14145 consensus_queue.cc:260] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } }
I20260929 16:29:41.563413 14145 ts_tablet_manager.cc:1428] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 16:29:41.565003 14146 ts_tablet_manager.cc:1428] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
W20260929 16:29:41.583901 14120 tablet.cc:2369] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:29:41.638662 14150 raft_consensus.cc:482] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:41.638931 14150 raft_consensus.cc:504] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } }
I20260929 16:29:41.640650 14150 leader_election.cc:288] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027), bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959)
I20260929 16:29:41.647890 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.641044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:34615 (local address 127.5.232.131:34027)
0929 16:29:41.641359 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:41.641368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:41.641801 (+ 433us) server_negotiation.cc:407] Connection header received
0929 16:29:41.642026 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:41.642035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:41.642142 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:41.642240 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:41.642943 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.643863 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.645964 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.646847 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.647019 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.647087 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:41.647476 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 16:29:41.647645 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 16:29:41.647997 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.640943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:34027 (local address 127.5.232.130:34615)
0929 16:29:41.641651 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:41.641667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:41.641859 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:41.642319 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:41.642322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:41.642349 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:41.642782 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.642796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.643996 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.644001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.645793 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.645806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.647094 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.647100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.647299 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.647341 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:29:41.647423 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:29:41.647570 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":66,"thread_start_us":339,"threads_started":1}
I20260929 16:29:41.648029 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.641984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:56455 (local address 127.5.232.129:44959)
0929 16:29:41.642214 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:41.642220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:41.642562 (+ 342us) server_negotiation.cc:407] Connection header received
0929 16:29:41.642703 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:41.642709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:41.642803 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:41.642901 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:41.643450 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.644298 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.646237 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.647092 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.647246 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.647315 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:41.647661 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 16:29:41.647806 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:29:41.648531 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.641825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:44959 (local address 127.5.232.130:56455)
0929 16:29:41.642454 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:41.642468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:41.642573 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:41.642939 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:41.642943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:41.642965 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:41.643305 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.643316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.644441 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.644444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.646069 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.646092 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.647339 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.647343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.647503 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.647534 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:29:41.647625 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:29:41.647813 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":5,"thread_start_us":288,"threads_started":1}
I20260929 16:29:41.649541 14074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195360cc753b449bb763eb9446a64cc2" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60a69e849ba4d708c87fa074398bd95" is_pre_election: true
I20260929 16:29:41.649751 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195360cc753b449bb763eb9446a64cc2" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460" is_pre_election: true
I20260929 16:29:41.650097 14074 raft_consensus.cc:2454] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 0.
I20260929 16:29:41.650350 13791 raft_consensus.cc:2454] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 0.
I20260929 16:29:41.650799 13861 leader_election.cc:302] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c703ebf25d24a3ebefd7163855efa42, e60a69e849ba4d708c87fa074398bd95; no voters:
I20260929 16:29:41.651182 14150 raft_consensus.cc:2781] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:41.651341 14150 raft_consensus.cc:482] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:41.651461 14150 raft_consensus.cc:3037] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:41.653672 14150 raft_consensus.cc:504] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } }
I20260929 16:29:41.654286 14150 leader_election.cc:288] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 election: Requested vote from peers e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027), bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959)
I20260929 16:29:41.654805 14074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195360cc753b449bb763eb9446a64cc2" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60a69e849ba4d708c87fa074398bd95"
I20260929 16:29:41.654875 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195360cc753b449bb763eb9446a64cc2" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460"
I20260929 16:29:41.654999 14074 raft_consensus.cc:3037] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:41.655143 13791 raft_consensus.cc:3037] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:41.657212 14074 raft_consensus.cc:2454] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 1.
I20260929 16:29:41.657335 13791 raft_consensus.cc:2454] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 1.
I20260929 16:29:41.657660 13861 leader_election.cc:302] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c703ebf25d24a3ebefd7163855efa42, e60a69e849ba4d708c87fa074398bd95; no voters:
I20260929 16:29:41.658007 14150 raft_consensus.cc:2781] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:41.658512 14150 raft_consensus.cc:686] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 LEADER]: Becoming Leader. State: Replica: 4c703ebf25d24a3ebefd7163855efa42, State: Running, Role: LEADER
I20260929 16:29:41.659068 14150 consensus_queue.cc:237] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } }
I20260929 16:29:41.663841 13633 catalog_manager.cc:5495] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130). New cstate: current_term: 1 leader_uuid: "4c703ebf25d24a3ebefd7163855efa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } health_report { overall_health: UNKNOWN } } }
I20260929 16:29:41.698669 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.693028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.5.232.130:43763)
0929 16:29:41.693255 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:41.693262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:41.693404 (+ 142us) server_negotiation.cc:407] Connection header received
0929 16:29:41.693686 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:41.693690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:41.693780 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:41.693973 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:41.694325 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.695147 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.696482 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.696893 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.697025 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.697499 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:41.698073 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:41.698246 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:41.698302 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 16:29:41.698382 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:29:41.706743 13791 raft_consensus.cc:1261] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Refusing update from remote peer 4c703ebf25d24a3ebefd7163855efa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:41.706858 14074 raft_consensus.cc:1261] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Refusing update from remote peer 4c703ebf25d24a3ebefd7163855efa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:41.707736 14155 consensus_queue.cc:1035] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:41.708089 14150 consensus_queue.cc:1035] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:41.722234 14160 mvcc.cc:205] Tried to move back new op lower bound from 7334704667461296128 to 7334704667279937536. Current Snapshot: MvccSnapshot[applied={T|T < 7334704667461296128}]
I20260929 16:29:41.729903 14159 mvcc.cc:205] Tried to move back new op lower bound from 7334704667461296128 to 7334704667279937536. Current Snapshot: MvccSnapshot[applied={T|T < 7334704667461296128}]
I20260929 16:29:41.740228 13633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:41.740504 13633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:41.744266 13633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8: Sending DeleteTablet for 3 replicas of tablet 195360cc753b449bb763eb9446a64cc2
I20260929 16:29:41.745553 13910 tablet_service.cc:1514] Processing DeleteTablet for tablet 195360cc753b449bb763eb9446a64cc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:41 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:39294
I20260929 16:29:41.745630 14049 tablet_service.cc:1514] Processing DeleteTablet for tablet 195360cc753b449bb763eb9446a64cc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:41 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:49768
I20260929 16:29:41.745826 13771 tablet_service.cc:1514] Processing DeleteTablet for tablet 195360cc753b449bb763eb9446a64cc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:41 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:47832
I20260929 16:29:41.750427 13633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:41.751320 13633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:41.755852 14170 tablet_replica.cc:330] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: stopping tablet replica
I20260929 16:29:41.758409 14170 raft_consensus.cc:2229] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:41.759367 14170 raft_consensus.cc:2258] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:41.762132 14168 tablet_replica.cc:330] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: stopping tablet replica
I20260929 16:29:41.762708 14168 raft_consensus.cc:2229] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:29:41.763433 13910 tablet_service.cc:1467] Processing CreateTablet for tablet c73eb363a0674955b7ea60843212a1a2 (DEFAULT_TABLE table=test-table [id=a991d295b8314b1395a4263c0f62cdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:41.763957 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73eb363a0674955b7ea60843212a1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:41.764202 14048 tablet_service.cc:1467] Processing CreateTablet for tablet c73eb363a0674955b7ea60843212a1a2 (DEFAULT_TABLE table=test-table [id=a991d295b8314b1395a4263c0f62cdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:41.764958 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73eb363a0674955b7ea60843212a1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:41.765182 13771 tablet_service.cc:1467] Processing CreateTablet for tablet c73eb363a0674955b7ea60843212a1a2 (DEFAULT_TABLE table=test-table [id=a991d295b8314b1395a4263c0f62cdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:41.765739 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73eb363a0674955b7ea60843212a1a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:41.771260 14168 raft_consensus.cc:2258] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:41.773455 14168 ts_tablet_manager.cc:1905] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:41.775125 14146 tablet_bootstrap.cc:492] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: Bootstrap starting.
I20260929 16:29:41.775801 14145 tablet_bootstrap.cc:492] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: Bootstrap starting.
I20260929 16:29:41.776878 14146 tablet_bootstrap.cc:654] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:41.777429 14145 tablet_bootstrap.cc:654] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:41.778035 14170 ts_tablet_manager.cc:1905] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:41.780297 14168 ts_tablet_manager.cc:1918] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:41.780490 14168 log.cc:1196] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/195360cc753b449bb763eb9446a64cc2
I20260929 16:29:41.781131 14168 ts_tablet_manager.cc:1939] T 195360cc753b449bb763eb9446a64cc2 P 4c703ebf25d24a3ebefd7163855efa42: Deleting consensus metadata
I20260929 16:29:41.781972 14169 tablet_replica.cc:330] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: stopping tablet replica
I20260929 16:29:41.782641 14169 raft_consensus.cc:2229] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:41.783023 14144 tablet_bootstrap.cc:492] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: Bootstrap starting.
I20260929 16:29:41.785000 14144 tablet_bootstrap.cc:654] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:41.790324 14169 raft_consensus.cc:2258] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:41.791257 14146 tablet_bootstrap.cc:492] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: No bootstrap required, opened a new log
I20260929 16:29:41.791428 14146 ts_tablet_manager.cc:1397] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260929 16:29:41.792307 14146 raft_consensus.cc:348] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.792532 14146 raft_consensus.cc:374] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:41.792626 14146 raft_consensus.cc:729] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad05bea81614f598e72f9e066d7b460, State: Initialized, Role: FOLLOWER
I20260929 16:29:41.792919 14146 consensus_queue.cc:260] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.794752 14169 ts_tablet_manager.cc:1905] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:41.795909 14146 ts_tablet_manager.cc:1428] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 16:29:41.800940 14170 ts_tablet_manager.cc:1918] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:41.801139 14170 log.cc:1196] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/195360cc753b449bb763eb9446a64cc2
I20260929 16:29:41.801730 14170 ts_tablet_manager.cc:1939] T 195360cc753b449bb763eb9446a64cc2 P bad05bea81614f598e72f9e066d7b460: Deleting consensus metadata
I20260929 16:29:41.803489 14145 tablet_bootstrap.cc:492] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: No bootstrap required, opened a new log
I20260929 16:29:41.803702 14145 ts_tablet_manager.cc:1397] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.012s
I20260929 16:29:41.804452 13618 catalog_manager.cc:4841] TS bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959): tablet 195360cc753b449bb763eb9446a64cc2 (table test-table [id=16408012499d4031b1d95419787bc13f]) successfully deleted
I20260929 16:29:41.804580 14145 raft_consensus.cc:348] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.804826 14145 raft_consensus.cc:374] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:41.804927 14145 raft_consensus.cc:729] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60a69e849ba4d708c87fa074398bd95, State: Initialized, Role: FOLLOWER
I20260929 16:29:41.805212 14145 consensus_queue.cc:260] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.805752 14144 tablet_bootstrap.cc:492] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: No bootstrap required, opened a new log
I20260929 16:29:41.805815 14145 ts_tablet_manager.cc:1428] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:29:41.805903 14144 ts_tablet_manager.cc:1397] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.003s
I20260929 16:29:41.806357 13620 catalog_manager.cc:4841] TS 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130:43763): tablet 195360cc753b449bb763eb9446a64cc2 (table test-table [id=16408012499d4031b1d95419787bc13f]) successfully deleted
I20260929 16:29:41.806738 14144 raft_consensus.cc:348] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.806957 14144 raft_consensus.cc:374] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:41.807047 14144 raft_consensus.cc:729] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c703ebf25d24a3ebefd7163855efa42, State: Initialized, Role: FOLLOWER
I20260929 16:29:41.807305 14144 consensus_queue.cc:260] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.807837 14144 ts_tablet_manager.cc:1428] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:29:41.810642 14169 ts_tablet_manager.cc:1918] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:41.810863 14169 log.cc:1196] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/195360cc753b449bb763eb9446a64cc2
I20260929 16:29:41.811440 14169 ts_tablet_manager.cc:1939] T 195360cc753b449bb763eb9446a64cc2 P e60a69e849ba4d708c87fa074398bd95: Deleting consensus metadata
I20260929 16:29:41.813122 13618 catalog_manager.cc:4841] TS e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027): tablet 195360cc753b449bb763eb9446a64cc2 (table test-table [id=16408012499d4031b1d95419787bc13f]) successfully deleted
I20260929 16:29:41.961108 14152 raft_consensus.cc:482] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:41.961431 14152 raft_consensus.cc:504] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.963182 14152 leader_election.cc:288] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130:43763)
I20260929 16:29:41.971534 14163 raft_consensus.cc:482] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:41.971760 14163 raft_consensus.cc:504] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.972162 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.963599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:44959 (local address 127.5.232.131:55541)
0929 16:29:41.964354 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:41.964381 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:41.964663 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:41.965303 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:41.965311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:41.965350 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:41.965870 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.965886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.967631 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.967638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.969628 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.969639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.971322 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.971328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.971503 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.971535 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:29:41.971650 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:29:41.971773 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":3523,"threads_started":1}
I20260929 16:29:41.972630 14163 leader_election.cc:288] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027)
I20260929 16:29:41.972780 14143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.963800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:55541 (local address 127.5.232.129:44959)
0929 16:29:41.964085 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:41.964092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:41.964490 (+ 398us) server_negotiation.cc:407] Connection header received
0929 16:29:41.964871 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:41.964880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:41.964982 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:41.965060 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:41.966092 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.967473 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.969784 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.971082 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.971253 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.971324 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:41.972371 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0929 16:29:41.972511 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260929 16:29:41.973592 14074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73eb363a0674955b7ea60843212a1a2" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60a69e849ba4d708c87fa074398bd95" is_pre_election: true
I20260929 16:29:41.973616 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73eb363a0674955b7ea60843212a1a2" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460" is_pre_election: true
I20260929 16:29:41.973857 14074 raft_consensus.cc:2454] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 0.
I20260929 16:29:41.973883 13791 raft_consensus.cc:2454] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 0.
I20260929 16:29:41.974040 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73eb363a0674955b7ea60843212a1a2" candidate_uuid: "e60a69e849ba4d708c87fa074398bd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460" is_pre_election: true
I20260929 16:29:41.974357 13861 leader_election.cc:302] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c703ebf25d24a3ebefd7163855efa42, e60a69e849ba4d708c87fa074398bd95; no voters:
I20260929 16:29:41.974378 13790 raft_consensus.cc:2454] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e60a69e849ba4d708c87fa074398bd95 in term 0.
I20260929 16:29:41.974808 14163 raft_consensus.cc:2781] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:41.974929 14163 raft_consensus.cc:482] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:41.975029 14163 raft_consensus.cc:3037] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:41.976248 14000 leader_election.cc:302] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bad05bea81614f598e72f9e066d7b460, e60a69e849ba4d708c87fa074398bd95; no voters:
I20260929 16:29:41.976689 14152 raft_consensus.cc:2781] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:41.976800 14152 raft_consensus.cc:482] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:41.976876 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.964354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:55071 (local address 127.5.232.130:43763)
0929 16:29:41.964558 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:41.964576 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:41.967543 (+ 2967us) server_negotiation.cc:407] Connection header received
0929 16:29:41.967802 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:41.967809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:41.967896 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:41.967987 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:41.968790 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.969928 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.972241 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.975723 (+ 3482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.975904 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.975987 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:41.976368 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 16:29:41.976520 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:29:41.976894 14152 raft_consensus.cc:3037] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:41.977612 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:41.963655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:43763 (local address 127.5.232.131:55071)
0929 16:29:41.967426 (+ 3771us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:41.967444 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:41.967642 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:41.968154 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:41.968158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:41.968181 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:41.968583 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.968594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.970072 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.970079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.972024 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:41.972050 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:41.975956 (+ 3906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:41.975964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:41.976163 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:41.976201 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:29:41.977117 (+ 916us) client_negotiation.cc:240] Negotiation successful
0929 16:29:41.977265 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3641,"thread_start_us":1561,"threads_started":1}
I20260929 16:29:41.978637 14163 raft_consensus.cc:504] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.978694 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73eb363a0674955b7ea60843212a1a2" candidate_uuid: "e60a69e849ba4d708c87fa074398bd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c703ebf25d24a3ebefd7163855efa42" is_pre_election: true
I20260929 16:29:41.979233 13930 raft_consensus.cc:2379] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e60a69e849ba4d708c87fa074398bd95 in current term 1: Already voted for candidate 4c703ebf25d24a3ebefd7163855efa42 in this term.
I20260929 16:29:41.980212 14074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73eb363a0674955b7ea60843212a1a2" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60a69e849ba4d708c87fa074398bd95"
I20260929 16:29:41.980512 14163 leader_election.cc:288] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 election: Requested vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027)
I20260929 16:29:41.980522 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73eb363a0674955b7ea60843212a1a2" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460"
I20260929 16:29:41.980749 13790 raft_consensus.cc:3037] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:41.982785 13790 raft_consensus.cc:2454] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 1.
I20260929 16:29:41.980235 14152 raft_consensus.cc:504] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.983206 13861 leader_election.cc:302] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c703ebf25d24a3ebefd7163855efa42, bad05bea81614f598e72f9e066d7b460; no voters:
I20260929 16:29:41.983525 14074 raft_consensus.cc:2379] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c703ebf25d24a3ebefd7163855efa42 in current term 1: Already voted for candidate e60a69e849ba4d708c87fa074398bd95 in this term.
I20260929 16:29:41.983795 14152 leader_election.cc:288] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [CANDIDATE]: Term 1 election: Requested vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130:43763)
I20260929 16:29:41.984357 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73eb363a0674955b7ea60843212a1a2" candidate_uuid: "e60a69e849ba4d708c87fa074398bd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c703ebf25d24a3ebefd7163855efa42"
I20260929 16:29:41.984397 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73eb363a0674955b7ea60843212a1a2" candidate_uuid: "e60a69e849ba4d708c87fa074398bd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460"
I20260929 16:29:41.984635 13930 raft_consensus.cc:2379] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e60a69e849ba4d708c87fa074398bd95 in current term 1: Already voted for candidate 4c703ebf25d24a3ebefd7163855efa42 in this term.
I20260929 16:29:41.984681 13790 raft_consensus.cc:2379] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e60a69e849ba4d708c87fa074398bd95 in current term 1: Already voted for candidate 4c703ebf25d24a3ebefd7163855efa42 in this term.
I20260929 16:29:41.984793 14163 raft_consensus.cc:2781] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:41.985198 14002 leader_election.cc:302] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e60a69e849ba4d708c87fa074398bd95; no voters: 4c703ebf25d24a3ebefd7163855efa42, bad05bea81614f598e72f9e066d7b460
I20260929 16:29:41.985296 14163 raft_consensus.cc:686] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 LEADER]: Becoming Leader. State: Replica: 4c703ebf25d24a3ebefd7163855efa42, State: Running, Role: LEADER
I20260929 16:29:41.985581 14152 raft_consensus.cc:2726] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 16:29:41.985697 14163 consensus_queue.cc:237] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:41.988575 13633 catalog_manager.cc:5495] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130). New cstate: current_term: 1 leader_uuid: "4c703ebf25d24a3ebefd7163855efa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } health_report { overall_health: UNKNOWN } } }
W20260929 16:29:42.034574 13842 tablet.cc:2369] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:29:42 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699378, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.190@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
W20260929 16:29:42.056528 13981 tablet.cc:2369] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:29:42.062126 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.043696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44087 (local address 127.0.0.1:60066)
0929 16:29:42.045970 (+ 2274us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:42.045996 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:42.046182 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:42.046715 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:42.046721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:42.047740 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:42.048023 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:42.048035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.049361 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.049369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:42.050937 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:42.050953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.051725 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.051733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:42.051920 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.052680 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:42.052710 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:42.055664 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:42.059048 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:42.059063 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:42.059072 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:42.059727 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:42.060084 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:42.060089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:42.060092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:42.060283 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:42.060896 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:42.060905 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:42.061098 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 16:29:42.061300 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 16:29:42.061656 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2082,"thread_start_us":1937,"threads_started":1}
I20260929 16:29:42.062175 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.043750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.5.232.190:44087)
0929 16:29:42.046408 (+ 2658us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:42.046413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:42.046434 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:29:42.046478 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:42.046481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:42.046533 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:42.046639 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:42.048187 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.049229 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.051098 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.051536 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.051643 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.055908 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:42.055913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:42.055916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:42.055946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:42.058896 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:42.059853 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:42.059857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:42.059859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:42.059969 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:42.060432 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:42.060435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:42.060437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:42.060726 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:42.060837 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:42.061487 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 16:29:42.061780 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":2546,"thread_start_us":2416,"threads_started":1}
I20260929 16:29:42.067683 13633 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:60066:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:29:42.068794 13633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:42.075726 13771 tablet_service.cc:1467] Processing CreateTablet for tablet fa6edd9b819540d8a11251cd22b8ed09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc63dc40cf084ea987e396fdbab4d548]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:29:42.075835 13910 tablet_service.cc:1467] Processing CreateTablet for tablet fa6edd9b819540d8a11251cd22b8ed09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc63dc40cf084ea987e396fdbab4d548]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:29:42.076336 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6edd9b819540d8a11251cd22b8ed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:42.076344 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6edd9b819540d8a11251cd22b8ed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:42.083782 14146 tablet_bootstrap.cc:492] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: Bootstrap starting.
I20260929 16:29:42.083812 14144 tablet_bootstrap.cc:492] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: Bootstrap starting.
I20260929 16:29:42.085230 14146 tablet_bootstrap.cc:654] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:42.085716 14144 tablet_bootstrap.cc:654] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:42.087142 14049 tablet_service.cc:1467] Processing CreateTablet for tablet fa6edd9b819540d8a11251cd22b8ed09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc63dc40cf084ea987e396fdbab4d548]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:29:42.087738 14049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6edd9b819540d8a11251cd22b8ed09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:42.090539 14146 tablet_bootstrap.cc:492] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: No bootstrap required, opened a new log
I20260929 16:29:42.090724 14146 ts_tablet_manager.cc:1397] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 16:29:42.091455 14146 raft_consensus.cc:348] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.091727 14146 raft_consensus.cc:374] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:42.091799 14146 raft_consensus.cc:729] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad05bea81614f598e72f9e066d7b460, State: Initialized, Role: FOLLOWER
I20260929 16:29:42.092036 14146 consensus_queue.cc:260] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.092515 14151 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.092845 14151 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:29:42.094405 14145 tablet_bootstrap.cc:492] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: Bootstrap starting.
I20260929 16:29:42.095031 14146 ts_tablet_manager.cc:1428] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:29:42.096201 14145 tablet_bootstrap.cc:654] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:42.097729 14144 tablet_bootstrap.cc:492] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: No bootstrap required, opened a new log
I20260929 16:29:42.097945 14144 ts_tablet_manager.cc:1397] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260929 16:29:42.098798 14144 raft_consensus.cc:348] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.099035 14144 raft_consensus.cc:374] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:42.099128 14144 raft_consensus.cc:729] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c703ebf25d24a3ebefd7163855efa42, State: Initialized, Role: FOLLOWER
I20260929 16:29:42.099364 14144 consensus_queue.cc:260] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.099979 14144 ts_tablet_manager.cc:1428] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:29:42.099947 14163 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.100238 14163 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:29:42.102496 14145 tablet_bootstrap.cc:492] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: No bootstrap required, opened a new log
I20260929 16:29:42.102686 14145 ts_tablet_manager.cc:1397] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260929 16:29:42.103312 14145 raft_consensus.cc:348] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.103549 14145 raft_consensus.cc:374] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:42.103624 14145 raft_consensus.cc:729] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60a69e849ba4d708c87fa074398bd95, State: Initialized, Role: FOLLOWER
I20260929 16:29:42.103847 14145 consensus_queue.cc:260] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.104264 14152 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.104527 14152 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:29:42.105032 14145 ts_tablet_manager.cc:1428] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:29:42.156996 14152 raft_consensus.cc:482] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:42.157239 14152 raft_consensus.cc:504] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.161923 14152 leader_election.cc:288] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130:43763)
I20260929 16:29:42.162408 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6edd9b819540d8a11251cd22b8ed09" candidate_uuid: "e60a69e849ba4d708c87fa074398bd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460" is_pre_election: true
I20260929 16:29:42.162690 13790 raft_consensus.cc:2454] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e60a69e849ba4d708c87fa074398bd95 in term 0.
I20260929 16:29:42.163200 14000 leader_election.cc:302] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bad05bea81614f598e72f9e066d7b460, e60a69e849ba4d708c87fa074398bd95; no voters:
I20260929 16:29:42.163537 14152 raft_consensus.cc:2781] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:42.163827 14152 raft_consensus.cc:482] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:42.163932 14152 raft_consensus.cc:3037] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:42.165910 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6edd9b819540d8a11251cd22b8ed09" candidate_uuid: "e60a69e849ba4d708c87fa074398bd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c703ebf25d24a3ebefd7163855efa42" is_pre_election: true
I20260929 16:29:42.166195 13930 raft_consensus.cc:2454] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e60a69e849ba4d708c87fa074398bd95 in term 0.
I20260929 16:29:42.168197 14152 raft_consensus.cc:504] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.169064 14152 leader_election.cc:288] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [CANDIDATE]: Term 1 election: Requested vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130:43763)
I20260929 16:29:42.170372 13930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6edd9b819540d8a11251cd22b8ed09" candidate_uuid: "e60a69e849ba4d708c87fa074398bd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c703ebf25d24a3ebefd7163855efa42"
I20260929 16:29:42.170594 13930 raft_consensus.cc:3037] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:42.173729 13930 raft_consensus.cc:2454] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e60a69e849ba4d708c87fa074398bd95 in term 1.
I20260929 16:29:42.174286 14002 leader_election.cc:302] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c703ebf25d24a3ebefd7163855efa42, e60a69e849ba4d708c87fa074398bd95; no voters:
I20260929 16:29:42.174613 14152 raft_consensus.cc:2781] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:42.174817 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6edd9b819540d8a11251cd22b8ed09" candidate_uuid: "e60a69e849ba4d708c87fa074398bd95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460"
I20260929 16:29:42.175057 13790 raft_consensus.cc:3037] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:42.177100 14152 raft_consensus.cc:686] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [term 1 LEADER]: Becoming Leader. State: Replica: e60a69e849ba4d708c87fa074398bd95, State: Running, Role: LEADER
I20260929 16:29:42.177829 14152 consensus_queue.cc:237] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.178125 13790 raft_consensus.cc:2454] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e60a69e849ba4d708c87fa074398bd95 in term 1.
I20260929 16:29:42.180174 14187 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: TxnStatusTablet state changed. Reason: New leader e60a69e849ba4d708c87fa074398bd95. Latest consensus state: current_term: 1 leader_uuid: "e60a69e849ba4d708c87fa074398bd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.180478 14187 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:29:42.181435 14189 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:29:42.181684 14189 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:29:42.182091 13634 catalog_manager.cc:5495] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 reported cstate change: term changed from 0 to 1, leader changed from <none> to e60a69e849ba4d708c87fa074398bd95 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "e60a69e849ba4d708c87fa074398bd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } health_report { overall_health: HEALTHY } } }
I20260929 16:29:42.218719 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.208633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:34027 (local address 127.0.0.1:49784)
0929 16:29:42.209245 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:42.209268 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:42.209500 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:42.211036 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:42.211044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:42.211060 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:29:42.211281 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:42.211296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.213298 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.213323 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:42.215330 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:42.215353 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.216410 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.216417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:42.216672 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.216779 (+ 107us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:29:42.217976 (+ 1197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:29:42.218001 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:29:42.218093 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:29:42.218299 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388}
I20260929 16:29:42.219771 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.209844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.5.232.131:34027)
0929 16:29:42.210468 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:42.210480 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:42.210523 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:29:42.210648 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:42.210653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:42.210787 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:42.210998 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:42.211445 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.213099 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.215549 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.216219 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.216379 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.217097 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:42.217862 (+ 765us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:42.218019 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:42.219194 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0929 16:29:42.219297 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":345,"threads_started":1}
I20260929 16:29:42.224332 13790 raft_consensus.cc:1261] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Refusing update from remote peer e60a69e849ba4d708c87fa074398bd95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:42.224376 13930 raft_consensus.cc:1261] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Refusing update from remote peer e60a69e849ba4d708c87fa074398bd95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:42.225199 14187 consensus_queue.cc:1035] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:42.225622 14152 consensus_queue.cc:1035] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:42.228972 14151 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: TxnStatusTablet state changed. Reason: New leader e60a69e849ba4d708c87fa074398bd95. Latest consensus state: current_term: 1 leader_uuid: "e60a69e849ba4d708c87fa074398bd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.229287 14151 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:29:42.233058 14152 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e60a69e849ba4d708c87fa074398bd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.233417 14152 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:29:42.236853 14151 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e60a69e849ba4d708c87fa074398bd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.237118 14151 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P bad05bea81614f598e72f9e066d7b460: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:29:42.249125 14163 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: TxnStatusTablet state changed. Reason: New leader e60a69e849ba4d708c87fa074398bd95. Latest consensus state: current_term: 1 leader_uuid: "e60a69e849ba4d708c87fa074398bd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.249507 14163 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:29:42.250711 14187 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e60a69e849ba4d708c87fa074398bd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.250982 14187 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P e60a69e849ba4d708c87fa074398bd95: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:29:42.257994 14163 tablet_replica.cc:439] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e60a69e849ba4d708c87fa074398bd95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } } }
I20260929 16:29:42.258329 14163 tablet_replica.cc:442] T fa6edd9b819540d8a11251cd22b8ed09 P 4c703ebf25d24a3ebefd7163855efa42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:29:42.259636 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.250561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:49305 (local address 127.5.232.131:34027)
0929 16:29:42.251238 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:42.251243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:42.251262 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:29:42.251322 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:42.251325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:42.251383 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:42.251498 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:42.252124 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.254654 (+ 2530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.256762 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.257686 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.257820 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.258815 (+ 995us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:42.258956 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:42.259051 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:42.259270 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 16:29:42.259357 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559}
I20260929 16:29:42.260406 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.249849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:34027 (local address 127.5.232.130:49305)
0929 16:29:42.250428 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:42.250450 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:42.250590 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:42.251516 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:42.251521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:42.251536 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:29:42.251892 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:42.251904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.254837 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.254844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:42.256586 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:42.256603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.257995 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.258002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:42.258189 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.258302 (+ 113us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:29:42.259121 (+ 819us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:29:42.259134 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 16:29:42.259830 (+ 696us) client_negotiation.cc:240] Negotiation successful
0929 16:29:42.260008 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":748,"threads_started":1}
I20260929 16:29:42.270159 14074 raft_consensus.cc:1261] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Refusing update from remote peer 4c703ebf25d24a3ebefd7163855efa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:42.270219 13790 raft_consensus.cc:1261] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Refusing update from remote peer 4c703ebf25d24a3ebefd7163855efa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:42.270897 14150 consensus_queue.cc:1035] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:42.271220 14163 consensus_queue.cc:1035] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:42.341300 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.325580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:35781 (local address 127.5.232.130:43763)
0929 16:29:42.325900 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:42.325906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:42.332297 (+ 6391us) server_negotiation.cc:407] Connection header received
0929 16:29:42.332570 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:42.332575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:42.332654 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:42.332710 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:42.333533 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.334641 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.339040 (+ 4399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.339698 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.339838 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.340490 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:42.340633 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:42.340741 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:42.340905 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:29:42.340995 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260929 16:29:42.342437 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.325569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:43763 (local address 127.5.232.131:35781)
0929 16:29:42.332194 (+ 6625us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:42.332215 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:42.332418 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:42.332909 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:42.332914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:42.332926 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:29:42.333305 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:42.333316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.336780 (+ 3464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.336788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:42.338790 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:42.338806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.339912 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.339919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:42.340155 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.340237 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:29:42.340740 (+ 503us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:29:42.340751 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:29:42.341851 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0929 16:29:42.342020 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4132,"thread_start_us":3997,"threads_started":1}
I20260929 16:29:42.380944 13633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:42.381207 13633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:42.384370 13633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8: Sending DeleteTablet for 3 replicas of tablet c73eb363a0674955b7ea60843212a1a2
I20260929 16:29:42.385250 13910 tablet_service.cc:1514] Processing DeleteTablet for tablet c73eb363a0674955b7ea60843212a1a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:42 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:39294
I20260929 16:29:42.385284 13771 tablet_service.cc:1514] Processing DeleteTablet for tablet c73eb363a0674955b7ea60843212a1a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:42 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:47832
I20260929 16:29:42.385918 14049 tablet_service.cc:1514] Processing DeleteTablet for tablet c73eb363a0674955b7ea60843212a1a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:42 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:49768
I20260929 16:29:42.401059 14209 tablet_replica.cc:330] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: stopping tablet replica
I20260929 16:29:42.401371 14209 raft_consensus.cc:2229] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:42.401760 14209 raft_consensus.cc:2258] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:42.402330 14208 tablet_replica.cc:330] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: stopping tablet replica
I20260929 16:29:42.402522 14208 raft_consensus.cc:2229] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:42.402765 14208 raft_consensus.cc:2258] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:42.403353 14209 ts_tablet_manager.cc:1905] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:42.403801 14208 ts_tablet_manager.cc:1905] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:42.410964 14208 ts_tablet_manager.cc:1918] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:29:42.411118 14208 log.cc:1196] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/c73eb363a0674955b7ea60843212a1a2
I20260929 16:29:42.411573 14208 ts_tablet_manager.cc:1939] T c73eb363a0674955b7ea60843212a1a2 P bad05bea81614f598e72f9e066d7b460: Deleting consensus metadata
I20260929 16:29:42.412711 13618 catalog_manager.cc:4841] TS bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959): tablet c73eb363a0674955b7ea60843212a1a2 (table test-table [id=a991d295b8314b1395a4263c0f62cdb8]) successfully deleted
I20260929 16:29:42.413367 14209 ts_tablet_manager.cc:1918] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:29:42.413519 14207 tablet_replica.cc:330] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: stopping tablet replica
I20260929 16:29:42.413697 14209 log.cc:1196] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/c73eb363a0674955b7ea60843212a1a2
I20260929 16:29:42.413767 14207 raft_consensus.cc:2229] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:29:42.414139 14207 raft_consensus.cc:2258] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:42.414353 14209 ts_tablet_manager.cc:1939] T c73eb363a0674955b7ea60843212a1a2 P e60a69e849ba4d708c87fa074398bd95: Deleting consensus metadata
I20260929 16:29:42.415242 14207 ts_tablet_manager.cc:1905] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:42.415521 13618 catalog_manager.cc:4841] TS e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027): tablet c73eb363a0674955b7ea60843212a1a2 (table test-table [id=a991d295b8314b1395a4263c0f62cdb8]) successfully deleted
I20260929 16:29:42.420507 14207 ts_tablet_manager.cc:1918] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:29:42.420658 14207 log.cc:1196] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/c73eb363a0674955b7ea60843212a1a2
I20260929 16:29:42.421066 14207 ts_tablet_manager.cc:1939] T c73eb363a0674955b7ea60843212a1a2 P 4c703ebf25d24a3ebefd7163855efa42: Deleting consensus metadata
I20260929 16:29:42.422267 13620 catalog_manager.cc:4841] TS 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130:43763): tablet c73eb363a0674955b7ea60843212a1a2 (table test-table [id=a991d295b8314b1395a4263c0f62cdb8]) successfully deleted
Sep 29 16:29:42 dist-test-slave-dssx krb5kdc[13589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:29:42 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:42 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:29:42.451042 14183 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:42.440450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.5.232.190:44087)
0929 16:29:42.440671 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:42.440678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:42.441064 (+ 386us) server_negotiation.cc:407] Connection header received
0929 16:29:42.441270 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:42.441280 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:42.441364 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:42.441496 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:42.442771 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.443508 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.444738 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.445167 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.445312 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.450863 (+ 5551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60068: BlockingRecv error: recv got EOF from 127.0.0.1:60068 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 16:29:42 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM
I20260929 16:29:42.470765 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.454659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.5.232.190:44087)
0929 16:29:42.454841 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:42.454848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:42.455294 (+ 446us) server_negotiation.cc:407] Connection header received
0929 16:29:42.455469 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:42.455473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:42.455534 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:42.455614 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:42.457124 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.457899 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.459718 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.460214 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.460316 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.464660 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:42.464665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:42.464668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:42.464710 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:42.467931 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:42.468733 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:42.468739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:42.468743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:42.468923 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:42.469265 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:42.469270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:42.469274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:42.469661 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:42.469932 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:42.470125 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 16:29:42.470357 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:29:42.474542 13633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:42.475286 13633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:42.481964 13771 tablet_service.cc:1467] Processing CreateTablet for tablet 711b3f490691448dbd8b46c75319b226 (DEFAULT_TABLE table=test-table [id=b7d1579131344c72afcf4b6efd48df3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:42.482431 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711b3f490691448dbd8b46c75319b226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:42.482362 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 711b3f490691448dbd8b46c75319b226 (DEFAULT_TABLE table=test-table [id=b7d1579131344c72afcf4b6efd48df3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:42.482728 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711b3f490691448dbd8b46c75319b226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:42.482715 14049 tablet_service.cc:1467] Processing CreateTablet for tablet 711b3f490691448dbd8b46c75319b226 (DEFAULT_TABLE table=test-table [id=b7d1579131344c72afcf4b6efd48df3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:42.483122 14049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711b3f490691448dbd8b46c75319b226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:42.488967 14144 tablet_bootstrap.cc:492] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: Bootstrap starting.
I20260929 16:29:42.489729 14145 tablet_bootstrap.cc:492] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: Bootstrap starting.
I20260929 16:29:42.489814 14146 tablet_bootstrap.cc:492] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: Bootstrap starting.
I20260929 16:29:42.490633 14144 tablet_bootstrap.cc:654] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:42.491300 14145 tablet_bootstrap.cc:654] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:42.491362 14146 tablet_bootstrap.cc:654] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:42.501916 14146 tablet_bootstrap.cc:492] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: No bootstrap required, opened a new log
I20260929 16:29:42.502527 14146 ts_tablet_manager.cc:1397] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20260929 16:29:42.502964 14145 tablet_bootstrap.cc:492] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: No bootstrap required, opened a new log
I20260929 16:29:42.503119 14145 ts_tablet_manager.cc:1397] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260929 16:29:42.503940 14145 raft_consensus.cc:348] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.504216 14145 raft_consensus.cc:374] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:42.504312 14145 raft_consensus.cc:729] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60a69e849ba4d708c87fa074398bd95, State: Initialized, Role: FOLLOWER
I20260929 16:29:42.504583 14145 consensus_queue.cc:260] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.505636 14146 raft_consensus.cc:348] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.506182 14146 raft_consensus.cc:374] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:42.506356 14146 raft_consensus.cc:729] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad05bea81614f598e72f9e066d7b460, State: Initialized, Role: FOLLOWER
I20260929 16:29:42.506731 14146 consensus_queue.cc:260] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.507510 14145 ts_tablet_manager.cc:1428] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 16:29:42.510176 14146 ts_tablet_manager.cc:1428] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 16:29:42.511195 14144 tablet_bootstrap.cc:492] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: No bootstrap required, opened a new log
I20260929 16:29:42.511372 14144 ts_tablet_manager.cc:1397] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20260929 16:29:42.512006 14144 raft_consensus.cc:348] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.512298 14144 raft_consensus.cc:374] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:42.512415 14144 raft_consensus.cc:729] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c703ebf25d24a3ebefd7163855efa42, State: Initialized, Role: FOLLOWER
I20260929 16:29:42.512666 14144 consensus_queue.cc:260] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.513125 14144 ts_tablet_manager.cc:1428] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:29:42.542095 14203 raft_consensus.cc:482] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:42.542336 14203 raft_consensus.cc:504] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.543126 14203 leader_election.cc:288] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027)
I20260929 16:29:42.543640 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711b3f490691448dbd8b46c75319b226" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460" is_pre_election: true
I20260929 16:29:42.543728 14074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711b3f490691448dbd8b46c75319b226" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60a69e849ba4d708c87fa074398bd95" is_pre_election: true
I20260929 16:29:42.543903 13790 raft_consensus.cc:2454] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 0.
I20260929 16:29:42.543972 14074 raft_consensus.cc:2454] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 0.
I20260929 16:29:42.544360 13861 leader_election.cc:302] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c703ebf25d24a3ebefd7163855efa42, bad05bea81614f598e72f9e066d7b460; no voters:
I20260929 16:29:42.544641 14203 raft_consensus.cc:2781] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:42.544785 14203 raft_consensus.cc:482] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:42.544878 14203 raft_consensus.cc:3037] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:42.548131 14203 raft_consensus.cc:504] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.548801 14203 leader_election.cc:288] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 election: Requested vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027)
I20260929 16:29:42.549299 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711b3f490691448dbd8b46c75319b226" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460"
I20260929 16:29:42.549331 14074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711b3f490691448dbd8b46c75319b226" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60a69e849ba4d708c87fa074398bd95"
I20260929 16:29:42.549561 13790 raft_consensus.cc:3037] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:42.549598 14074 raft_consensus.cc:3037] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:42.552670 13790 raft_consensus.cc:2454] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 1.
I20260929 16:29:42.552822 14074 raft_consensus.cc:2454] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 1.
I20260929 16:29:42.553118 13861 leader_election.cc:302] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c703ebf25d24a3ebefd7163855efa42, bad05bea81614f598e72f9e066d7b460; no voters:
I20260929 16:29:42.553647 14203 raft_consensus.cc:2781] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:42.553900 14203 raft_consensus.cc:686] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 LEADER]: Becoming Leader. State: Replica: 4c703ebf25d24a3ebefd7163855efa42, State: Running, Role: LEADER
I20260929 16:29:42.554170 14203 consensus_queue.cc:237] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.557039 13634 catalog_manager.cc:5495] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130). New cstate: current_term: 1 leader_uuid: "4c703ebf25d24a3ebefd7163855efa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } health_report { overall_health: UNKNOWN } } }
I20260929 16:29:42.579147 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.574112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.5.232.130:43763)
0929 16:29:42.574298 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:42.574305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:42.574465 (+ 160us) server_negotiation.cc:407] Connection header received
0929 16:29:42.574714 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:42.574719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:42.574785 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:42.574885 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:42.575259 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.575976 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.577293 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.577811 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.577960 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.578425 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:42.578548 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:42.578652 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:42.578770 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 16:29:42.578853 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:29:42.582027 14074 raft_consensus.cc:1261] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Refusing update from remote peer 4c703ebf25d24a3ebefd7163855efa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:42.582250 13790 raft_consensus.cc:1261] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Refusing update from remote peer 4c703ebf25d24a3ebefd7163855efa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:42.582631 14150 consensus_queue.cc:1035] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:42.582895 14203 consensus_queue.cc:1035] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:42.592119 14193 mvcc.cc:205] Tried to move back new op lower bound from 7334704671049392128 to 7334704670944575488. Current Snapshot: MvccSnapshot[applied={T|T < 7334704671049392128}]
I20260929 16:29:42.601812 13633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:42.602068 13633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:42.605052 13633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8: Sending DeleteTablet for 3 replicas of tablet 711b3f490691448dbd8b46c75319b226
I20260929 16:29:42.605973 13910 tablet_service.cc:1514] Processing DeleteTablet for tablet 711b3f490691448dbd8b46c75319b226 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:42 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:39294
I20260929 16:29:42.606009 14049 tablet_service.cc:1514] Processing DeleteTablet for tablet 711b3f490691448dbd8b46c75319b226 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:42 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:49768
I20260929 16:29:42.606256 14207 tablet_replica.cc:330] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: stopping tablet replica
I20260929 16:29:42.606262 14209 tablet_replica.cc:330] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: stopping tablet replica
I20260929 16:29:42.606472 14209 raft_consensus.cc:2229] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:42.606459 14207 raft_consensus.cc:2229] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:29:42.606694 14209 raft_consensus.cc:2258] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:42.606775 14207 raft_consensus.cc:2258] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:42.607259 13771 tablet_service.cc:1514] Processing DeleteTablet for tablet 711b3f490691448dbd8b46c75319b226 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:42 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:47832
I20260929 16:29:42.607535 14208 tablet_replica.cc:330] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: stopping tablet replica
I20260929 16:29:42.607722 14208 raft_consensus.cc:2229] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:42.607869 14209 ts_tablet_manager.cc:1905] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:42.607929 14208 raft_consensus.cc:2258] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:42.608338 13633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:42.608970 13633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:42.609064 14208 ts_tablet_manager.cc:1905] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:42.610697 14207 ts_tablet_manager.cc:1905] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:42.614746 14209 ts_tablet_manager.cc:1918] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:42.614884 14209 log.cc:1196] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/711b3f490691448dbd8b46c75319b226
I20260929 16:29:42.615525 14209 ts_tablet_manager.cc:1939] T 711b3f490691448dbd8b46c75319b226 P e60a69e849ba4d708c87fa074398bd95: Deleting consensus metadata
I20260929 16:29:42.615541 14208 ts_tablet_manager.cc:1918] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:42.616765 13618 catalog_manager.cc:4841] TS e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027): tablet 711b3f490691448dbd8b46c75319b226 (table test-table [id=b7d1579131344c72afcf4b6efd48df3a]) successfully deleted
I20260929 16:29:42.617209 14207 ts_tablet_manager.cc:1918] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:42.617316 14207 log.cc:1196] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/711b3f490691448dbd8b46c75319b226
I20260929 16:29:42.617688 14207 ts_tablet_manager.cc:1939] T 711b3f490691448dbd8b46c75319b226 P 4c703ebf25d24a3ebefd7163855efa42: Deleting consensus metadata
I20260929 16:29:42.618686 13620 catalog_manager.cc:4841] TS 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130:43763): tablet 711b3f490691448dbd8b46c75319b226 (table test-table [id=b7d1579131344c72afcf4b6efd48df3a]) successfully deleted
I20260929 16:29:42.615675 14208 log.cc:1196] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/711b3f490691448dbd8b46c75319b226
I20260929 16:29:42.620956 14208 ts_tablet_manager.cc:1939] T 711b3f490691448dbd8b46c75319b226 P bad05bea81614f598e72f9e066d7b460: Deleting consensus metadata
I20260929 16:29:42.622145 13618 catalog_manager.cc:4841] TS bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959): tablet 711b3f490691448dbd8b46c75319b226 (table test-table [id=b7d1579131344c72afcf4b6efd48df3a]) successfully deleted
I20260929 16:29:42.622709 13771 tablet_service.cc:1467] Processing CreateTablet for tablet 3653dd472caa46958bb2eb7ab2a75bec (DEFAULT_TABLE table=test-table [id=a4c0ceb0fc47422e99a3f3cc3fc0d183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:42.623083 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3653dd472caa46958bb2eb7ab2a75bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:42.623010 14049 tablet_service.cc:1467] Processing CreateTablet for tablet 3653dd472caa46958bb2eb7ab2a75bec (DEFAULT_TABLE table=test-table [id=a4c0ceb0fc47422e99a3f3cc3fc0d183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:42.623011 13910 tablet_service.cc:1467] Processing CreateTablet for tablet 3653dd472caa46958bb2eb7ab2a75bec (DEFAULT_TABLE table=test-table [id=a4c0ceb0fc47422e99a3f3cc3fc0d183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:42.623402 13910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3653dd472caa46958bb2eb7ab2a75bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:42.623404 14049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3653dd472caa46958bb2eb7ab2a75bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:42.629714 14146 tablet_bootstrap.cc:492] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: Bootstrap starting.
I20260929 16:29:42.631279 14144 tablet_bootstrap.cc:492] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: Bootstrap starting.
I20260929 16:29:42.631403 14146 tablet_bootstrap.cc:654] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:42.632452 14145 tablet_bootstrap.cc:492] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: Bootstrap starting.
I20260929 16:29:42.632683 14144 tablet_bootstrap.cc:654] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:42.634274 14145 tablet_bootstrap.cc:654] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:42.634934 14144 tablet_bootstrap.cc:492] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: No bootstrap required, opened a new log
I20260929 16:29:42.635051 14144 ts_tablet_manager.cc:1397] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260929 16:29:42.635813 14144 raft_consensus.cc:348] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.636087 14144 raft_consensus.cc:374] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:42.636176 14144 raft_consensus.cc:729] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c703ebf25d24a3ebefd7163855efa42, State: Initialized, Role: FOLLOWER
I20260929 16:29:42.636416 14144 consensus_queue.cc:260] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.637020 14144 ts_tablet_manager.cc:1428] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:29:42.637843 14146 tablet_bootstrap.cc:492] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: No bootstrap required, opened a new log
I20260929 16:29:42.637997 14146 ts_tablet_manager.cc:1397] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20260929 16:29:42.638675 14146 raft_consensus.cc:348] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.638895 14146 raft_consensus.cc:374] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:42.638950 14146 raft_consensus.cc:729] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad05bea81614f598e72f9e066d7b460, State: Initialized, Role: FOLLOWER
I20260929 16:29:42.639161 14146 consensus_queue.cc:260] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.639891 14146 ts_tablet_manager.cc:1428] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:29:42.641113 14145 tablet_bootstrap.cc:492] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: No bootstrap required, opened a new log
I20260929 16:29:42.641278 14145 ts_tablet_manager.cc:1397] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 16:29:42.642175 14145 raft_consensus.cc:348] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.642431 14145 raft_consensus.cc:374] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:42.642503 14145 raft_consensus.cc:729] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60a69e849ba4d708c87fa074398bd95, State: Initialized, Role: FOLLOWER
I20260929 16:29:42.642714 14145 consensus_queue.cc:260] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.643280 14145 ts_tablet_manager.cc:1428] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:29:42.675942 14203 raft_consensus.cc:482] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:42.676175 14203 raft_consensus.cc:504] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.676965 14203 leader_election.cc:288] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027)
I20260929 16:29:42.677557 14074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3653dd472caa46958bb2eb7ab2a75bec" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60a69e849ba4d708c87fa074398bd95" is_pre_election: true
I20260929 16:29:42.677579 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3653dd472caa46958bb2eb7ab2a75bec" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460" is_pre_election: true
I20260929 16:29:42.677863 14074 raft_consensus.cc:2454] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 0.
I20260929 16:29:42.677862 13790 raft_consensus.cc:2454] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 0.
I20260929 16:29:42.678375 13861 leader_election.cc:302] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c703ebf25d24a3ebefd7163855efa42, e60a69e849ba4d708c87fa074398bd95; no voters:
I20260929 16:29:42.678663 14203 raft_consensus.cc:2781] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:42.678799 14203 raft_consensus.cc:482] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:42.678913 14203 raft_consensus.cc:3037] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:42.680951 14203 raft_consensus.cc:504] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.681559 14203 leader_election.cc:288] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 election: Requested vote from peers bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959), e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027)
I20260929 16:29:42.682189 14074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3653dd472caa46958bb2eb7ab2a75bec" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e60a69e849ba4d708c87fa074398bd95"
I20260929 16:29:42.682219 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3653dd472caa46958bb2eb7ab2a75bec" candidate_uuid: "4c703ebf25d24a3ebefd7163855efa42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad05bea81614f598e72f9e066d7b460"
I20260929 16:29:42.682432 14074 raft_consensus.cc:3037] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:42.682447 13790 raft_consensus.cc:3037] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:42.685634 14074 raft_consensus.cc:2454] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 1.
I20260929 16:29:42.685775 13790 raft_consensus.cc:2454] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c703ebf25d24a3ebefd7163855efa42 in term 1.
I20260929 16:29:42.686251 13861 leader_election.cc:302] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c703ebf25d24a3ebefd7163855efa42, bad05bea81614f598e72f9e066d7b460; no voters:
I20260929 16:29:42.686578 14203 raft_consensus.cc:2781] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:42.686750 14203 raft_consensus.cc:686] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 1 LEADER]: Becoming Leader. State: Replica: 4c703ebf25d24a3ebefd7163855efa42, State: Running, Role: LEADER
I20260929 16:29:42.687036 14203 consensus_queue.cc:237] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } }
I20260929 16:29:42.689851 13634 catalog_manager.cc:5495] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130). New cstate: current_term: 1 leader_uuid: "4c703ebf25d24a3ebefd7163855efa42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c703ebf25d24a3ebefd7163855efa42" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 } health_report { overall_health: UNKNOWN } } }
I20260929 16:29:42.718793 14074 raft_consensus.cc:1261] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Refusing update from remote peer 4c703ebf25d24a3ebefd7163855efa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:42.718997 13790 raft_consensus.cc:1261] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Refusing update from remote peer 4c703ebf25d24a3ebefd7163855efa42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:42.719417 14203 consensus_queue.cc:1035] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e60a69e849ba4d708c87fa074398bd95" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 34027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:42.719841 14150 consensus_queue.cc:1035] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad05bea81614f598e72f9e066d7b460" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 44959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:42.770920 13634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:42.771183 13634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:42.774127 13634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6516f04fa06244f4aae5d7ae404035b8: Sending DeleteTablet for 3 replicas of tablet 3653dd472caa46958bb2eb7ab2a75bec
I20260929 16:29:42.774951 13771 tablet_service.cc:1514] Processing DeleteTablet for tablet 3653dd472caa46958bb2eb7ab2a75bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:42 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:47832
I20260929 16:29:42.775013 13910 tablet_service.cc:1514] Processing DeleteTablet for tablet 3653dd472caa46958bb2eb7ab2a75bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:42 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:39294
I20260929 16:29:42.775276 14207 tablet_replica.cc:330] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: stopping tablet replica
I20260929 16:29:42.775411 14208 tablet_replica.cc:330] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: stopping tablet replica
I20260929 16:29:42.775431 14207 raft_consensus.cc:2229] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:29:42.775544 14208 raft_consensus.cc:2229] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:42.775689 14208 raft_consensus.cc:2258] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:42.775741 14207 raft_consensus.cc:2258] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:42.776196 14049 tablet_service.cc:1514] Processing DeleteTablet for tablet 3653dd472caa46958bb2eb7ab2a75bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:42 UTC) from {username='oryx', principal='oryx/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:49768
I20260929 16:29:42.776520 14209 tablet_replica.cc:330] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: stopping tablet replica
I20260929 16:29:42.776697 14209 raft_consensus.cc:2229] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:42.776785 14208 ts_tablet_manager.cc:1905] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:42.776878 14209 raft_consensus.cc:2258] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:42.777956 14207 ts_tablet_manager.cc:1905] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:42.781312 14208 ts_tablet_manager.cc:1918] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:29:42.781490 14208 log.cc:1196] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/3653dd472caa46958bb2eb7ab2a75bec
I20260929 16:29:42.781962 14208 ts_tablet_manager.cc:1939] T 3653dd472caa46958bb2eb7ab2a75bec P bad05bea81614f598e72f9e066d7b460: Deleting consensus metadata
I20260929 16:29:42.783066 14209 ts_tablet_manager.cc:1905] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:42.783095 13618 catalog_manager.cc:4841] TS bad05bea81614f598e72f9e066d7b460 (127.5.232.129:44959): tablet 3653dd472caa46958bb2eb7ab2a75bec (table test-table [id=a4c0ceb0fc47422e99a3f3cc3fc0d183]) successfully deleted
I20260929 16:29:42.783577 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 13705
I20260929 16:29:42.787362 14207 ts_tablet_manager.cc:1918] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:29:42.787536 14207 log.cc:1196] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/3653dd472caa46958bb2eb7ab2a75bec
I20260929 16:29:42.788043 14207 ts_tablet_manager.cc:1939] T 3653dd472caa46958bb2eb7ab2a75bec P 4c703ebf25d24a3ebefd7163855efa42: Deleting consensus metadata
I20260929 16:29:42.789402 13620 catalog_manager.cc:4841] TS 4c703ebf25d24a3ebefd7163855efa42 (127.5.232.130:43763): tablet 3653dd472caa46958bb2eb7ab2a75bec (table test-table [id=a4c0ceb0fc47422e99a3f3cc3fc0d183]) successfully deleted
I20260929 16:29:42.791347 14209 ts_tablet_manager.cc:1918] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:29:42.791501 14209 log.cc:1196] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/3653dd472caa46958bb2eb7ab2a75bec
I20260929 16:29:42.792063 14209 ts_tablet_manager.cc:1939] T 3653dd472caa46958bb2eb7ab2a75bec P e60a69e849ba4d708c87fa074398bd95: Deleting consensus metadata
I20260929 16:29:42.793056 13618 catalog_manager.cc:4841] TS e60a69e849ba4d708c87fa074398bd95 (127.5.232.131:34027): tablet 3653dd472caa46958bb2eb7ab2a75bec (table test-table [id=a4c0ceb0fc47422e99a3f3cc3fc0d183]) successfully deleted
Sep 29 16:29:42 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.129@KRBTEST.COM
I20260929 16:29:42.806505 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.784770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.5.232.129:44959)
0929 16:29:42.792009 (+ 7239us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:42.792013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:42.792030 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:29:42.792074 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:42.792077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:42.792177 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:42.792281 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:42.793762 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.794570 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.795805 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.796204 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.796293 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.800083 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:42.800097 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:42.800113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:42.800152 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:42.803725 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:42.804382 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:42.804392 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:42.804396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:42.804546 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:42.804884 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:42.804889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:42.804893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:42.805353 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:42.805676 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:42.805894 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 16:29:42.806109 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7147,"thread_start_us":7075,"threads_started":1}
I20260929 16:29:42.807211 13836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:42.902746 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 13705
I20260929 16:29:42.920858 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 13845
Sep 29 16:29:42 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.130@KRBTEST.COM
I20260929 16:29:42.935884 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:42.921623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39328 (local address 127.5.232.130:43763)
0929 16:29:42.921807 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:42.921811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:42.921828 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:29:42.921943 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:42.921947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:42.922010 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:42.922092 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:42.923231 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.923933 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.925123 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:42.925681 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:42.925781 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:42.929892 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:42.929907 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:42.929924 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:42.929964 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:42.932842 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:42.933769 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:42.933778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:42.933781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:42.933913 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:42.934308 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:42.934312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:42.934314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:42.934681 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:42.934983 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:42.935327 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 16:29:42.935562 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:29:42.936666 13974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:43.060184 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 13845
I20260929 16:29:43.082644 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 13984
Sep 29 16:29:43 dist-test-slave-dssx krb5kdc[13589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.232.131@KRBTEST.COM
I20260929 16:29:43.097695 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:43.083420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.5.232.131:34027)
0929 16:29:43.083871 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:43.083875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:43.083890 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:29:43.083943 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:43.083946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:43.084002 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:43.084093 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:43.085447 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:43.086138 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:43.087373 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:43.087758 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:43.087852 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:43.092120 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:43.092134 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:43.092151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:43.092196 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:43.095079 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:43.095825 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:43.095832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:43.095835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:43.095971 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:43.096301 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:43.096304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:43.096306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:43.096627 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:43.096865 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:43.097085 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 16:29:43.097261 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260929 16:29:43.098349 14114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:43.219270 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 13984
I20260929 16:29:43.240854 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 13602
I20260929 16:29:43.242090 13674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:43.359239 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 13602
2026-09-29T16:29:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6508 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:43 dist-test-slave-dssx krb5kdc[14245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:43 dist-test-slave-dssx krb5kdc[14245](info): set up 2 sockets
Sep 29 16:29:43 dist-test-slave-dssx krb5kdc[14245](info): commencing operation
krb5kdc: starting...
W20260929 16:29:44.454571 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:29:44 dist-test-slave-dssx krb5kdc[14245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:29:44.680590 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:41977
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:41977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:45.003511 14258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:45.003824 14258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:45.003916 14258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:45.011248 14258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:45.011368 14258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:45.011415 14258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:45.011466 14258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:45.011505 14258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:45.021546 14258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:41977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:41977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:45.022861 14258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:45.024178 14258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:45.035738 14263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:45.035740 14264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:45.035741 14266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:45.036558 14258 server_base.cc:1034] running on GCE node
I20260929 16:29:45.037282 14258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:45.038753 14258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:45.039961 14258 hybrid_clock.cc:648] HybridClock initialized: now 1790699385039919 us; error 34 us; skew 500 ppm
I20260929 16:29:45.040325 14258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T16:29:45Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1747 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:45 dist-test-slave-dssx krb5kdc[14272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:45 dist-test-slave-dssx krb5kdc[14272](info): set up 2 sockets
Sep 29 16:29:45 dist-test-slave-dssx krb5kdc[14272](info): commencing operation
krb5kdc: starting...
W20260929 16:29:46.206285 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:29:46 dist-test-slave-dssx krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:29:46.429016 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40541
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:46.754663 14285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:46.755052 14285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:46.755221 14285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:46.762471 14285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:46.762640 14285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:46.762755 14285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:46.762873 14285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:46.762962 14285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:46.773026 14285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:40541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:46.774544 14285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:46.775974 14285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:46.788266 14293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:46.788378 14291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:46.788304 14290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:46.790789 14285 server_base.cc:1034] running on GCE node
I20260929 16:29:46.791622 14285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:46.793131 14285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:46.794374 14285 hybrid_clock.cc:648] HybridClock initialized: now 1790699386794299 us; error 89 us; skew 500 ppm
I20260929 16:29:46.794742 14285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:46 dist-test-slave-dssx krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:46.801862 14285 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:29:46.803556 14285 webserver.cc:460] Webserver started at http://127.5.232.190:39369/ using document root <none> and password file <none>
I20260929 16:29:46.804401 14285 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:46.804557 14285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:46.804906 14285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:46.808079 14285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "4f56918987204d5998a630f4eb20f5e8"
format_stamp: "Formatted at 2026-09-29 16:29:46 on dist-test-slave-dssx"
server_key: "89beebe31f3b4b3a31272a60f4e1dbf8"
server_key_iv: "4d3da457faba4fb0e3b7bff423b1ce33"
server_key_version: "encryptionkey@0"
I20260929 16:29:46.809108 14285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "4f56918987204d5998a630f4eb20f5e8"
format_stamp: "Formatted at 2026-09-29 16:29:46 on dist-test-slave-dssx"
server_key: "89beebe31f3b4b3a31272a60f4e1dbf8"
server_key_iv: "4d3da457faba4fb0e3b7bff423b1ce33"
server_key_version: "encryptionkey@0"
I20260929 16:29:46.814877 14285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:29:46.818547 14300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:46.819998 14285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:46.820190 14285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "4f56918987204d5998a630f4eb20f5e8"
format_stamp: "Formatted at 2026-09-29 16:29:46 on dist-test-slave-dssx"
server_key: "89beebe31f3b4b3a31272a60f4e1dbf8"
server_key_iv: "4d3da457faba4fb0e3b7bff423b1ce33"
server_key_version: "encryptionkey@0"
I20260929 16:29:46.820399 14285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:46.845985 14285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:46.850589 14285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:46.850939 14285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:46.869539 14285 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:40541
I20260929 16:29:46.869531 14351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:40541 every 8 connection(s)
I20260929 16:29:46.871028 14285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:46.874835 14352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:46.880689 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 14285
I20260929 16:29:46.880957 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:29:46.881783 6050 external_mini_cluster.cc:1428] Setting key a394c1c9351161101b0d004adecbf1d2
I20260929 16:29:46.886343 14352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: Bootstrap starting.
I20260929 16:29:46.890168 14352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:46.891379 14352 log.cc:824] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:46.894600 14352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: No bootstrap required, opened a new log
Sep 29 16:29:46 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:46.902702 14352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } }
I20260929 16:29:46.903090 14352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:46.903216 14352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f56918987204d5998a630f4eb20f5e8, State: Initialized, Role: FOLLOWER
I20260929 16:29:46.903851 14352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } }
I20260929 16:29:46.904127 14352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:46.904289 14352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:46.904479 14352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:46.905418 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:46.883961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.5.232.190:40541)
0929 16:29:46.884830 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:46.884842 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:46.884884 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:29:46.886693 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:46.886715 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:46.887112 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:46.887441 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:46.889337 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:46.890775 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:46.892448 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:46.893166 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:46.893312 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:46.897880 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:46.897888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:46.897903 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:46.897942 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:46.901091 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:46.901892 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:46.901906 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:46.901917 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:46.902083 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:46.902457 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:46.902464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:46.902467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:46.902980 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:46.903266 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:46.903708 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 16:29:46.904185 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":428,"threads_started":1}
I20260929 16:29:46.906968 14352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } }
I20260929 16:29:46.907470 14352 leader_election.cc:302] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f56918987204d5998a630f4eb20f5e8; no voters:
I20260929 16:29:46.908121 14352 leader_election.cc:288] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:46.908228 14357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:46.908851 14357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 1 LEADER]: Becoming Leader. State: Replica: 4f56918987204d5998a630f4eb20f5e8, State: Running, Role: LEADER
I20260929 16:29:46.909407 14357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } }
I20260929 16:29:46.910349 14352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:46.912170 14359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f56918987204d5998a630f4eb20f5e8. Latest consensus state: current_term: 1 leader_uuid: "4f56918987204d5998a630f4eb20f5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } } }
I20260929 16:29:46.912552 14359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:46.914639 14358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f56918987204d5998a630f4eb20f5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } } }
I20260929 16:29:46.915031 14358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:46.916114 14365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:46.921487 14365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:46.930771 14365 catalog_manager.cc:1347] Generated new cluster ID: cc04becf9960461eb13e03cd8bb7d38f
I20260929 16:29:46.930905 14365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:46.943123 14365 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:46.945022 14365 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:46.953157 14365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: Generated new TSK 0
I20260929 16:29:46.954136 14365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:29:47.003038 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40541
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:47.335814 14378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:47.336098 14378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:47.336265 14378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:47.343493 14378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:47.343596 14378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:47.343778 14378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:47.353574 14378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:47.354966 14378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:47.356252 14378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:47.370175 14384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:47.370223 14383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:47.370255 14386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:47.371297 14378 server_base.cc:1034] running on GCE node
I20260929 16:29:47.371925 14378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:47.372925 14378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:47.374132 14378 hybrid_clock.cc:648] HybridClock initialized: now 1790699387374094 us; error 39 us; skew 500 ppm
I20260929 16:29:47.374455 14378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:47 dist-test-slave-dssx krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699387, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:47.377754 14378 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:29:47.379343 14378 webserver.cc:460] Webserver started at http://127.5.232.129:38291/ using document root <none> and password file <none>
I20260929 16:29:47.380055 14378 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:47.380179 14378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:47.380497 14378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:47.383507 14378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "a74a1781217342d1a87da7ae15aa74f0"
format_stamp: "Formatted at 2026-09-29 16:29:47 on dist-test-slave-dssx"
server_key: "f7763e16a98a17f74917a8dee2325fb1"
server_key_iv: "f8e8b508b0b0ce294d188b3801512082"
server_key_version: "encryptionkey@0"
I20260929 16:29:47.384299 14378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "a74a1781217342d1a87da7ae15aa74f0"
format_stamp: "Formatted at 2026-09-29 16:29:47 on dist-test-slave-dssx"
server_key: "f7763e16a98a17f74917a8dee2325fb1"
server_key_iv: "f8e8b508b0b0ce294d188b3801512082"
server_key_version: "encryptionkey@0"
I20260929 16:29:47.389467 14378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:29:47.393143 14393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.394616 14378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:29:47.394807 14378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "a74a1781217342d1a87da7ae15aa74f0"
format_stamp: "Formatted at 2026-09-29 16:29:47 on dist-test-slave-dssx"
server_key: "f7763e16a98a17f74917a8dee2325fb1"
server_key_iv: "f8e8b508b0b0ce294d188b3801512082"
server_key_version: "encryptionkey@0"
I20260929 16:29:47.395006 14378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:47.414074 14378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:47.418493 14378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:47.418792 14378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:47.419751 14378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:47.421137 14378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:47.421211 14378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.421329 14378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:47.421411 14378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.455351 14378 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:35155
I20260929 16:29:47.455463 14505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:35155 every 8 connection(s)
I20260929 16:29:47.456674 14378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:47.465224 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 14378
I20260929 16:29:47.465484 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:47.466125 6050 external_mini_cluster.cc:1428] Setting key dd5c143c83a03ddd633d82f4c818759b
Sep 29 16:29:47 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699387, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:47.478112 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.459234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:41645 (local address 127.5.232.190:40541)
0929 16:29:47.459693 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:47.459698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:47.460491 (+ 793us) server_negotiation.cc:407] Connection header received
0929 16:29:47.462421 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:47.462426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:47.462516 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:47.462635 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:47.464120 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:47.464932 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:47.466831 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:47.467270 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:47.467401 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:47.471337 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:47.471346 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:47.471353 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:47.471397 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:47.474794 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:47.475491 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:47.475498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:47.475503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:47.475634 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:47.476137 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:47.476140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:47.476142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:47.476385 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:47.476486 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:47.477350 (+ 864us) server_negotiation.cc:299] Negotiation successful
0929 16:29:47.477620 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260929 16:29:47.478734 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.459654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:41645)
0929 16:29:47.460334 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:47.460376 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:47.462142 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:47.462874 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:47.462883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:47.463395 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:47.463953 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:47.463969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:47.465107 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:47.465111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:47.466626 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:47.466663 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:47.467482 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:47.467487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:47.467682 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:47.468431 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:47.468467 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:47.471100 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:47.474934 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:47.474941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:47.474955 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:47.475355 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:47.475750 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:47.475757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:47.475763 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:47.476022 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:47.476508 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:47.476519 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:47.476956 (+ 437us) client_negotiation.cc:769] Sending connection context
0929 16:29:47.477181 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 16:29:47.477634 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":259,"threads_started":1}
I20260929 16:29:47.480317 14506 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40541
I20260929 16:29:47.480618 14506 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:47.481329 14506 heartbeater.cc:511] Master 127.5.232.190:40541 requested a full tablet report, sending...
I20260929 16:29:47.483350 14317 ts_manager.cc:194] Registered new tserver with Master: a74a1781217342d1a87da7ae15aa74f0 (127.5.232.129:35155)
I20260929 16:29:47.484792 14317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:41645
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:29:47.514925 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40541
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:47.849313 14512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:47.849612 14512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:47.849706 14512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:47.857053 14512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:47.857159 14512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:47.857267 14512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:47.866784 14512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:47.867987 14512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:47.869333 14512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:47.881865 14518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:47.882098 14517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:47.882218 14520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:47.883514 14512 server_base.cc:1034] running on GCE node
I20260929 16:29:47.884184 14512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:47.885157 14512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:47.886379 14512 hybrid_clock.cc:648] HybridClock initialized: now 1790699387886324 us; error 51 us; skew 500 ppm
I20260929 16:29:47.886698 14512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:47 dist-test-slave-dssx krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699387, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:47.889801 14512 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:29:47.891306 14512 webserver.cc:460] Webserver started at http://127.5.232.130:46627/ using document root <none> and password file <none>
I20260929 16:29:47.891987 14512 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:47.892127 14512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:47.892436 14512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:47.895323 14512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "63cd0b41a92845779879a32663d2cda3"
format_stamp: "Formatted at 2026-09-29 16:29:47 on dist-test-slave-dssx"
server_key: "394d5af8ef9f4f1c3e9ac4d5fd940872"
server_key_iv: "c016914364b22ddf2fdb97b36d084d04"
server_key_version: "encryptionkey@0"
I20260929 16:29:47.896099 14512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "63cd0b41a92845779879a32663d2cda3"
format_stamp: "Formatted at 2026-09-29 16:29:47 on dist-test-slave-dssx"
server_key: "394d5af8ef9f4f1c3e9ac4d5fd940872"
server_key_iv: "c016914364b22ddf2fdb97b36d084d04"
server_key_version: "encryptionkey@0"
I20260929 16:29:47.901455 14512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:29:47.904922 14527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.906404 14512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:29:47.906584 14512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "63cd0b41a92845779879a32663d2cda3"
format_stamp: "Formatted at 2026-09-29 16:29:47 on dist-test-slave-dssx"
server_key: "394d5af8ef9f4f1c3e9ac4d5fd940872"
server_key_iv: "c016914364b22ddf2fdb97b36d084d04"
server_key_version: "encryptionkey@0"
I20260929 16:29:47.906790 14512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:47.919016 14512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:47.923038 14512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:47.923331 14512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:47.924373 14512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:47.925879 14512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:47.925983 14512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.926136 14512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:47.926208 14512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.959196 14512 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:44823
I20260929 16:29:47.959246 14639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:44823 every 8 connection(s)
I20260929 16:29:47.960574 14512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:47.966238 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 14512
I20260929 16:29:47.966444 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:47.966986 6050 external_mini_cluster.cc:1428] Setting key 136770d2c5b5653614b0eeffd7be2258
Sep 29 16:29:47 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699387, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:47.983795 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.963279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:40285 (local address 127.5.232.190:40541)
0929 16:29:47.963455 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:47.963459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:47.964732 (+ 1273us) server_negotiation.cc:407] Connection header received
0929 16:29:47.967101 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:47.967110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:47.967199 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:47.967292 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:47.969269 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:47.970026 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:47.972656 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:47.973090 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:47.973186 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:47.977630 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:47.977635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:47.977639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:47.977673 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:47.980622 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:47.981415 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:47.981423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:47.981427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:47.981560 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:47.982018 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:47.982025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:47.982028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:47.982306 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:47.982478 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:47.983192 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 16:29:47.983437 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:29:47.984959 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.963844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:40285)
0929 16:29:47.964550 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:47.964595 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:47.966730 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:47.967622 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:47.967640 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:47.968327 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:47.969080 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:47.969106 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:47.970198 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:47.970207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:47.972493 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:47.972514 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:47.973331 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:47.973338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:47.973687 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:47.974458 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:47.974500 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:47.977307 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:47.980785 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:47.980794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:47.980821 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:47.981257 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:47.981684 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:47.981689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:47.981693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:47.981903 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:47.982452 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:47.982461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:47.982823 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 16:29:47.983113 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 16:29:47.983670 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":271,"threads_started":1}
I20260929 16:29:47.986541 14640 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40541
I20260929 16:29:47.986914 14640 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:47.987944 14640 heartbeater.cc:511] Master 127.5.232.190:40541 requested a full tablet report, sending...
I20260929 16:29:47.989756 14317 ts_manager.cc:194] Registered new tserver with Master: 63cd0b41a92845779879a32663d2cda3 (127.5.232.130:44823)
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
I20260929 16:29:47.990649 14317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:40285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:29:48.018298 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40541
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:48.342826 14645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:48.343127 14645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:48.343291 14645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:48.350687 14645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:48.350806 14645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:48.351015 14645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:48.361150 14645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:48.362528 14645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:48.363931 14645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:48.378311 14650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:48.378789 14651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:48.380198 14653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:48.380821 14645 server_base.cc:1034] running on GCE node
I20260929 16:29:48.381618 14645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:48.382537 14645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:48.383738 14645 hybrid_clock.cc:648] HybridClock initialized: now 1790699388383687 us; error 42 us; skew 500 ppm
I20260929 16:29:48.384075 14645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:48 dist-test-slave-dssx krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699388, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:48.387382 14645 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:29:48.388928 14645 webserver.cc:460] Webserver started at http://127.5.232.131:33503/ using document root <none> and password file <none>
I20260929 16:29:48.389719 14645 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:48.389822 14645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:48.390089 14645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:48.392976 14645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "a39d8ffffea64ab2a1a1d38f474571b7"
format_stamp: "Formatted at 2026-09-29 16:29:48 on dist-test-slave-dssx"
server_key: "30d412433d260a4f14babd40261d51ba"
server_key_iv: "119a7b58b8849fcf3f9f9a04177e91cb"
server_key_version: "encryptionkey@0"
I20260929 16:29:48.393817 14645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "a39d8ffffea64ab2a1a1d38f474571b7"
format_stamp: "Formatted at 2026-09-29 16:29:48 on dist-test-slave-dssx"
server_key: "30d412433d260a4f14babd40261d51ba"
server_key_iv: "119a7b58b8849fcf3f9f9a04177e91cb"
server_key_version: "encryptionkey@0"
I20260929 16:29:48.399369 14645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:29:48.403147 14660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:48.404526 14645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:29:48.404695 14645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "a39d8ffffea64ab2a1a1d38f474571b7"
format_stamp: "Formatted at 2026-09-29 16:29:48 on dist-test-slave-dssx"
server_key: "30d412433d260a4f14babd40261d51ba"
server_key_iv: "119a7b58b8849fcf3f9f9a04177e91cb"
server_key_version: "encryptionkey@0"
I20260929 16:29:48.404920 14645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:48.421041 14645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:48.425908 14645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:48.426223 14645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:48.427271 14645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:48.428712 14645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:48.428790 14645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:48.428922 14645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:48.428982 14645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:48.463565 14645 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:42415
I20260929 16:29:48.463658 14772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:42415 every 8 connection(s)
I20260929 16:29:48.464907 14645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:48.469583 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 14645
I20260929 16:29:48.469799 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:48.470377 6050 external_mini_cluster.cc:1428] Setting key 1afe3869170c20653e90976a0c377b90
Sep 29 16:29:48 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699388, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:29:48.486315 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.467553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:54599 (local address 127.5.232.190:40541)
0929 16:29:48.467781 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:48.467788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:48.469124 (+ 1336us) server_negotiation.cc:407] Connection header received
0929 16:29:48.471822 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:48.471828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:48.471973 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:48.472061 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:48.474066 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.474816 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.476391 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.476787 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.476890 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:48.480299 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:48.480304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:48.480307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:48.480340 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:48.482762 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.483443 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.483447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.483449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.483567 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.484036 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.484042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.484045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.484442 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:48.484608 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:48.485674 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0929 16:29:48.485910 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 16:29:48.487448 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.468158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:54599)
0929 16:29:48.468934 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:48.468983 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:48.471458 (+ 2475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:48.472388 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:48.472397 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:48.473045 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:48.473907 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:48.473928 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.474953 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.474957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:48.476254 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:48.476266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.476995 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.477000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:48.477203 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:48.477913 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:48.477936 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:48.480093 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:48.482898 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:48.482906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:48.482920 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:48.483333 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:48.483659 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:48.483662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:48.483664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:48.483875 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:48.484716 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:48.484727 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:48.485187 (+ 460us) client_negotiation.cc:769] Sending connection context
0929 16:29:48.485534 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 16:29:48.486157 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":323,"threads_started":1}
I20260929 16:29:48.487663 14506 heartbeater.cc:503] Master 127.5.232.190:40541 was elected leader, sending a full tablet report...
I20260929 16:29:48.488915 14773 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40541
I20260929 16:29:48.489228 14773 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:48.490006 14773 heartbeater.cc:511] Master 127.5.232.190:40541 requested a full tablet report, sending...
I20260929 16:29:48.491349 14317 ts_manager.cc:194] Registered new tserver with Master: a39d8ffffea64ab2a1a1d38f474571b7 (127.5.232.131:42415)
I20260929 16:29:48.492126 14317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:54599
I20260929 16:29:48.499331 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab.
Entry for principal oryx/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab.
WARNING: no policy specified for oryx/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab.
Entry for principal oryx/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab.
WARNING: no policy specified for oryx/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab.
Entry for principal oryx/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab.
I20260929 16:29:48.633847 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 14378
Sep 29 16:29:48 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:29:48.649703 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.634995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58462 (local address 127.5.232.129:35155)
0929 16:29:48.635618 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:48.635626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:48.635653 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:29:48.635714 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:48.635721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:48.635932 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:48.636081 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:48.637463 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.638504 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.639772 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.640352 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.640459 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:48.644414 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:48.644425 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:48.644440 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:48.644473 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:48.647274 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.647960 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.647967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.647969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.648083 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.648380 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.648383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.648385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.648695 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:48.648907 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:48.649155 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 16:29:48.649342 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":325,"threads_started":1}
I20260929 16:29:48.651011 14501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:48.735387 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 14378
I20260929 16:29:48.751376 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 14512
Sep 29 16:29:48 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:29:48.767056 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.752254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.5.232.130:44823)
0929 16:29:48.752750 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:48.752756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:48.752780 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:29:48.752846 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:48.752851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:48.753048 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:48.753218 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:48.754545 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.755519 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.756784 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.757337 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.757610 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:48.761976 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:48.761987 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:48.762000 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:48.762032 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:48.764818 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.765419 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.765432 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.765434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.765548 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.765858 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.765861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.765863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.766176 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:48.766395 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:48.766594 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 16:29:48.766778 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":252,"threads_started":1}
I20260929 16:29:48.768318 14635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:48.850224 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 14512
I20260929 16:29:48.866428 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 14645
Sep 29 16:29:48 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:29:48.884526 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.867343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.5.232.131:42415)
0929 16:29:48.867917 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:48.867924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:48.867951 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:29:48.868010 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:48.868016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:48.868226 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:48.868393 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:48.870085 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.871105 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.872868 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.873531 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.873637 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:48.878877 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:48.878888 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:48.878904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:48.878938 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:48.881739 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.882467 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.882475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.882478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.882595 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.882972 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.882978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.882981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.883317 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:48.883642 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:48.883942 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 16:29:48.884202 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":266,"threads_started":1}
I20260929 16:29:48.885836 14768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:48.967455 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 14645
I20260929 16:29:48.982950 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 14285
I20260929 16:29:48.984102 14347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:29:49.084779 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 14285
I20260929 16:29:49.105202 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40541
--webserver_interface=127.5.232.190
--webserver_port=39369
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:40541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:49.430512 14790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:49.430881 14790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:49.431054 14790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:49.438557 14790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:49.438725 14790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:49.438840 14790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:49.438947 14790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:49.439034 14790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:49.449182 14790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:40541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=39369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:49.450670 14790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:49.452140 14790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:49.464525 14795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:49.464466 14798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:49.464546 14796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:49.465703 14790 server_base.cc:1034] running on GCE node
I20260929 16:29:49.466593 14790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:49.468099 14790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:49.469352 14790 hybrid_clock.cc:648] HybridClock initialized: now 1790699389469302 us; error 67 us; skew 500 ppm
I20260929 16:29:49.469779 14790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:49 dist-test-slave-dssx krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699389, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:49.473047 14790 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:29:49.474777 14790 webserver.cc:460] Webserver started at http://127.5.232.190:39369/ using document root <none> and password file <none>
I20260929 16:29:49.475539 14790 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:49.475706 14790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:49.481479 14790 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:29:49.484457 14805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:49.486106 14790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:49.486296 14790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "4f56918987204d5998a630f4eb20f5e8"
format_stamp: "Formatted at 2026-09-29 16:29:46 on dist-test-slave-dssx"
server_key: "89beebe31f3b4b3a31272a60f4e1dbf8"
server_key_iv: "4d3da457faba4fb0e3b7bff423b1ce33"
server_key_version: "encryptionkey@0"
I20260929 16:29:49.487284 14790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:49.521049 14790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:49.525583 14790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:49.525861 14790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:49.544427 14790 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:40541
I20260929 16:29:49.544448 14856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:40541 every 8 connection(s)
I20260929 16:29:49.545975 14790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:49.547472 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 14790
I20260929 16:29:49.548425 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:35155
--local_ip_for_outbound_sockets=127.5.232.129
--tserver_master_addrs=127.5.232.190:40541
--webserver_port=38291
--webserver_interface=127.5.232.129
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.232.129
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:29:49.555809 14857 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:29:49.559186 14857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: Bootstrap starting.
I20260929 16:29:49.578444 14857 log.cc:824] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:49.587234 14857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:29:49.587754 14857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: Bootstrap complete.
I20260929 16:29:49.597168 14857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } }
I20260929 16:29:49.598099 14857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f56918987204d5998a630f4eb20f5e8, State: Initialized, Role: FOLLOWER
I20260929 16:29:49.598783 14857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } }
I20260929 16:29:49.599040 14857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:49.599160 14857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:49.599356 14857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:29:49.603068 14857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } }
I20260929 16:29:49.603561 14857 leader_election.cc:302] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f56918987204d5998a630f4eb20f5e8; no voters:
I20260929 16:29:49.604193 14857 leader_election.cc:288] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:29:49.604327 14861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:29:49.605445 14861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [term 2 LEADER]: Becoming Leader. State: Replica: 4f56918987204d5998a630f4eb20f5e8, State: Running, Role: LEADER
I20260929 16:29:49.606082 14861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } }
I20260929 16:29:49.606453 14857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:49.608628 14862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4f56918987204d5998a630f4eb20f5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } } }
I20260929 16:29:49.609082 14862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:49.613456 14867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:49.613358 14863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f56918987204d5998a630f4eb20f5e8. Latest consensus state: current_term: 2 leader_uuid: "4f56918987204d5998a630f4eb20f5e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f56918987204d5998a630f4eb20f5e8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40541 } } }
I20260929 16:29:49.613860 14863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:49.619704 14867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:49.621570 14867 catalog_manager.cc:1259] Loaded cluster ID: cc04becf9960461eb13e03cd8bb7d38f
I20260929 16:29:49.621764 14867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:49.627081 14867 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:49.629089 14867 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4f56918987204d5998a630f4eb20f5e8: Loaded TSK: 0
I20260929 16:29:49.631124 14867 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:29:49.927568 14859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:49.927954 14859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:49.928150 14859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:49.937198 14859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:49.937342 14859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:49.937572 14859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:29:49.948048 14859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.129.keytab
--principal=oryx/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:35155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=38291
--tserver_master_addrs=127.5.232.190:40541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:49.949484 14859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:49.950819 14859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:49.966015 14883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:49.965989 14884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:49.966279 14886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:49.967525 14859 server_base.cc:1034] running on GCE node
I20260929 16:29:49.968219 14859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:49.969275 14859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:49.970494 14859 hybrid_clock.cc:648] HybridClock initialized: now 1790699389970439 us; error 50 us; skew 500 ppm
I20260929 16:29:49.970822 14859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:49 dist-test-slave-dssx krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699389, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:49.974112 14859 init.cc:377] Logged in from keytab as oryx/127.5.232.129@KRBTEST.COM (short username oryx)
I20260929 16:29:49.975736 14859 webserver.cc:460] Webserver started at http://127.5.232.129:38291/ using document root <none> and password file <none>
I20260929 16:29:49.976481 14859 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:49.976619 14859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:49.982234 14859 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:29:49.985195 14893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:49.986726 14859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:29:49.986892 14859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "a74a1781217342d1a87da7ae15aa74f0"
format_stamp: "Formatted at 2026-09-29 16:29:47 on dist-test-slave-dssx"
server_key: "f7763e16a98a17f74917a8dee2325fb1"
server_key_iv: "f8e8b508b0b0ce294d188b3801512082"
server_key_version: "encryptionkey@0"
I20260929 16:29:49.987866 14859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:50.010493 14859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:50.014873 14859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:50.015152 14859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:50.016120 14859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:50.017506 14859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:50.017611 14859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:50.017738 14859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:50.017819 14859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:50.050489 14859 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:35155
I20260929 16:29:50.050541 15005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:35155 every 8 connection(s)
I20260929 16:29:50.052018 14859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:50.054204 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 14859
I20260929 16:29:50.054813 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:44823
--local_ip_for_outbound_sockets=127.5.232.130
--tserver_master_addrs=127.5.232.190:40541
--webserver_port=46627
--webserver_interface=127.5.232.130
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.232.130
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:29:50 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:50 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:29:50.396795 15010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:50.397085 15010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:50.397238 15010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:50.404455 15010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:50.404569 15010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:50.404786 15010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:29:50.414871 15010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.130.keytab
--principal=oryx/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:44823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=46627
--tserver_master_addrs=127.5.232.190:40541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:50.416234 15010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:50.417675 15010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:50.436107 15017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:50.436292 15019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:50.436836 15016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:50.437880 15010 server_base.cc:1034] running on GCE node
I20260929 16:29:50.438544 15010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:50.439579 15010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:50.440992 15010 hybrid_clock.cc:648] HybridClock initialized: now 1790699390440907 us; error 85 us; skew 500 ppm
I20260929 16:29:50.441324 15010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:50 dist-test-slave-dssx krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699390, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:50.444774 15010 init.cc:377] Logged in from keytab as oryx/127.5.232.130@KRBTEST.COM (short username oryx)
I20260929 16:29:50.446481 15010 webserver.cc:460] Webserver started at http://127.5.232.130:46627/ using document root <none> and password file <none>
I20260929 16:29:50.447187 15010 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:50.447335 15010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:50.452805 15010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:29:50.455601 15026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:50.457017 15010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:50.457187 15010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "63cd0b41a92845779879a32663d2cda3"
format_stamp: "Formatted at 2026-09-29 16:29:47 on dist-test-slave-dssx"
server_key: "394d5af8ef9f4f1c3e9ac4d5fd940872"
server_key_iv: "c016914364b22ddf2fdb97b36d084d04"
server_key_version: "encryptionkey@0"
I20260929 16:29:50.458248 15010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:50.471750 15010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:50.476126 15010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:50.476398 15010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:50.477461 15010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:50.478885 15010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:50.478968 15010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:50.479105 15010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:50.479166 15010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:50.524400 15010 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:44823
I20260929 16:29:50.524446 15138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:44823 every 8 connection(s)
I20260929 16:29:50.525961 15010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:50.530273 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 15010
I20260929 16:29:50.531013 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:42415
--local_ip_for_outbound_sockets=127.5.232.131
--tserver_master_addrs=127.5.232.190:40541
--webserver_port=33503
--webserver_interface=127.5.232.131
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.232.131
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:29:50 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:50 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:29:51.163856 15009 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.097s user 0.000s sys 0.012s
W20260929 16:29:51.165913 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:50.055228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:54125 (local address 127.5.232.190:40541)
0929 16:29:50.056026 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:50.056045 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:50.056632 (+ 587us) server_negotiation.cc:407] Connection header received
0929 16:29:50.059045 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:50.059067 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:50.059556 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:50.059951 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:50.061334 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:50.063450 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:50.064987 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:50.066163 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:50.066354 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.165078 (+1098724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:54125: BlockingRecv error: recv got EOF from 127.5.232.129:54125 (error 108)
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":285,"threads_started":1}
I20260929 16:29:51.165902 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:50.055356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:54125)
0929 16:29:50.056426 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:50.056477 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:50.058425 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:50.060084 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:50.060092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:50.060587 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:50.061129 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:50.061146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:50.063603 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:50.063607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:50.064838 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:50.064849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:50.066373 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:50.066378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:50.066569 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:50.067262 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:50.067289 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:51.164671 (+1097382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":615,"threads_started":1}
W20260929 16:29:51.167045 15006 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:40541 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.232.190:40541: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:51.177251 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:51.167785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:41977)
0929 16:29:51.168571 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:51.168596 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:51.168748 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:51.169232 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:51.169237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:51.169650 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:51.169812 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.169821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.170843 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.170849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.171776 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.171788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.172530 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.172535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.172700 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.173334 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:51.173353 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:51.176922 (+ 3569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":1386,"threads_started":1}
W20260929 16:29:51.177433 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:51.168164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:41977 (local address 127.5.232.190:40541)
0929 16:29:51.168331 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:51.168337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:51.168665 (+ 328us) server_negotiation.cc:407] Connection header received
0929 16:29:51.168879 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:51.168886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:51.168991 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:51.169090 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:51.169976 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.170685 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.171941 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.172332 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.172473 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.177140 (+ 4667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:41977: BlockingRecv error: recv got EOF from 127.5.232.129:41977 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:51.187318 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:51.178440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:53641)
0929 16:29:51.178755 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:51.178769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:51.178915 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:51.179268 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:51.179274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:51.179602 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:51.179758 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.179766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.180798 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.180803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.181853 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.181867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.182806 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.182810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.182957 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.183576 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:51.183595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:51.187023 (+ 3428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238}
W20260929 16:29:51.187525 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:51.178508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:53641 (local address 127.5.232.190:40541)
0929 16:29:51.178662 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:51.178666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:51.178847 (+ 181us) server_negotiation.cc:407] Connection header received
0929 16:29:51.179037 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:51.179041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:51.179122 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:51.179230 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:51.179916 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.180631 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.182008 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.182430 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.182579 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.187277 (+ 4698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:53641: BlockingRecv error: recv got EOF from 127.5.232.129:53641 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 16:29:51.187935 15006 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:29:51.218861 15143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:51.219213 15143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:51.219386 15143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:51.227031 15143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:51.227202 15143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:51.227383 15143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:29:51.237635 15143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/krb5kdc/oryx_127.5.232.131.keytab
--principal=oryx/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:42415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=33503
--tserver_master_addrs=127.5.232.190:40541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:29:51.239046 15143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:51.240479 15143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:51.259114 15150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:51.259065 15151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:51.259917 15153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:51.260375 15143 server_base.cc:1034] running on GCE node
I20260929 16:29:51.261042 15143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:51.262202 15143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:51.263423 15143 hybrid_clock.cc:648] HybridClock initialized: now 1790699391263345 us; error 68 us; skew 500 ppm
I20260929 16:29:51.263758 15143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699391, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:51.267467 15143 init.cc:377] Logged in from keytab as oryx/127.5.232.131@KRBTEST.COM (short username oryx)
I20260929 16:29:51.269199 15143 webserver.cc:460] Webserver started at http://127.5.232.131:33503/ using document root <none> and password file <none>
I20260929 16:29:51.270015 15143 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:51.270176 15143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:51.275969 15143 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:29:51.279341 15160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:51.281322 15143 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:29:51.281554 15143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "a39d8ffffea64ab2a1a1d38f474571b7"
format_stamp: "Formatted at 2026-09-29 16:29:48 on dist-test-slave-dssx"
server_key: "30d412433d260a4f14babd40261d51ba"
server_key_iv: "119a7b58b8849fcf3f9f9a04177e91cb"
server_key_version: "encryptionkey@0"
I20260929 16:29:51.282604 15143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:51.294584 15143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:51.298936 15143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:51.299224 15143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:51.300231 15143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:51.301713 15143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:51.301834 15143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:51.301961 15143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:51.302047 15143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:51.344517 15143 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:42415
I20260929 16:29:51.344573 15272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:42415 every 8 connection(s)
I20260929 16:29:51.346125 15143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:51.351692 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 15143
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:51.393152 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:51.352840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.5.232.190:40541)
0929 16:29:51.369440 (+ 16600us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:51.369457 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:51.369479 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:29:51.369538 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:51.369543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:51.370187 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:51.370316 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:51.373286 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.374743 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.377543 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.378298 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.378450 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.383536 (+ 5086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:51.383583 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:51.383614 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:51.383659 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:51.387013 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:51.388192 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:51.388209 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:51.388225 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:51.388559 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:51.389312 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:51.389321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:51.389328 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:51.390378 (+ 1050us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:51.391054 (+ 676us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:51.391769 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 16:29:51.392508 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16415,"thread_start_us":16282,"threads_started":1}
W20260929 16:29:51.788972 15142 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.247s user 0.000s sys 0.021s
W20260929 16:29:51.790305 15141 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:50.529014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:37029 (local address 127.5.232.190:40541)
0929 16:29:50.529508 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:50.529513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:50.531907 (+ 2394us) server_negotiation.cc:407] Connection header received
0929 16:29:50.534514 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:50.534524 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:50.534654 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:50.534832 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:50.536979 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:50.538723 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:50.540554 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:50.540960 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:50.541114 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.789983 (+1248869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:37029: BlockingRecv error: recv got EOF from 127.5.232.130:37029 (error 108)
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
I20260929 16:29:51.790972 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:50.530085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:37029)
0929 16:29:50.531352 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:50.531404 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:50.534155 (+ 2751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:50.535109 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:50.535125 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:50.535915 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:50.536757 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:50.536786 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:50.538965 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:50.538971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:50.540387 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:50.540399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:50.541205 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:50.541214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:50.541520 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:50.542210 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:50.542233 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:51.789791 (+1247558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":403,"threads_started":1}
W20260929 16:29:51.791968 15139 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:40541 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.232.190:40541: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:51.808550 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:51.792864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:40905)
0929 16:29:51.793225 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:51.793255 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:51.793478 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:51.793843 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:51.793851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:51.794540 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:51.794848 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.794866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.795898 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.795906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.798333 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.798358 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.799332 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.799338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.799515 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.804168 (+ 4653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:51.804187 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:51.808251 (+ 4064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187}
W20260929 16:29:51.808696 15141 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:51.792838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:40905 (local address 127.5.232.190:40541)
0929 16:29:51.792997 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:51.793001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:51.793342 (+ 341us) server_negotiation.cc:407] Connection header received
0929 16:29:51.793614 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:51.793618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:51.793680 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:51.793761 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:51.795009 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.795730 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.798625 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.799151 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.799279 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.808441 (+ 9162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:40905: BlockingRecv error: recv got EOF from 127.5.232.130:40905 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:51 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:29:51.822614 15141 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:51.809884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:41355 (local address 127.5.232.190:40541)
0929 16:29:51.810050 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:51.810054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:51.810182 (+ 128us) server_negotiation.cc:407] Connection header received
0929 16:29:51.810405 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:51.810409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:51.810484 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:51.810567 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:51.811726 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.812645 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.815365 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.815756 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.815852 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.822420 (+ 6568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:41355: BlockingRecv error: recv got EOF from 127.5.232.130:41355 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:29:51.822711 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:51.809762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:41355)
0929 16:29:51.810090 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:51.810116 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:51.810274 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:51.810659 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:51.810667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:51.811255 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:51.811569 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.811586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.812837 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.812852 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.815186 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.815209 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.815933 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.815943 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.816227 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.816969 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:51.817033 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:51.822274 (+ 5241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174}
W20260929 16:29:51.823117 15139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 16:29:52 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:52 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:29:52.200454 15141 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:52.188745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:46477 (local address 127.5.232.190:40541)
0929 16:29:52.188912 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:52.188917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:52.189416 (+ 499us) server_negotiation.cc:407] Connection header received
0929 16:29:52.189621 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:52.189626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:52.189717 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:52.189825 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:52.190661 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.191672 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.193161 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.193682 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.193822 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:52.200273 (+ 6451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:46477: BlockingRecv error: recv got EOF from 127.5.232.129:46477 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:29:52.200583 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:52.188721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:46477)
0929 16:29:52.189268 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:52.189283 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:52.189488 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:52.189866 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:52.189870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:52.190325 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:52.190516 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:52.190532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.191802 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.191808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:52.193013 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:52.193025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.193872 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.193877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:52.194058 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:52.194774 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:52.194814 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:52.200078 (+ 5264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":342,"threads_started":1}
W20260929 16:29:52.481804 15275 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.114s user 0.000s sys 0.015s
W20260929 16:29:52.483006 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:51.349435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:51227 (local address 127.5.232.190:40541)
0929 16:29:51.349601 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:51.349606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:51.351806 (+ 2200us) server_negotiation.cc:407] Connection header received
0929 16:29:51.355261 (+ 3455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:51.355271 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:51.355388 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:51.355494 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:51.358597 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.360390 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.364074 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.365104 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.365243 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:52.482699 (+1117456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:51227: BlockingRecv error: recv got EOF from 127.5.232.131:51227 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:29:52.483610 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:51.350486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:51227)
0929 16:29:51.351619 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:51.351676 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:51.354816 (+ 3140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:51.356181 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:51.356199 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:51.357198 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:51.358307 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.358339 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.360709 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.360718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.363806 (+ 3088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:51.363831 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:51.365761 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:51.365772 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:51.366220 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:51.367975 (+ 1755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:51.368004 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:52.482547 (+1114543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":391,"threads_started":1}
W20260929 16:29:52.484426 15273 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:40541 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.232.190:40541: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Sep 29 16:29:52 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:52 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:52.493508 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:52.485009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:39023)
0929 16:29:52.485191 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:52.485210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:52.485324 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:52.485717 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:52.485722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:52.486080 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:52.486229 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:52.486237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.487250 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.487257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:52.488211 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:52.488221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.488958 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.488965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:52.489126 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:52.489693 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:52.489710 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:52.493196 (+ 3486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:29:52.493660 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:52.485059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:39023 (local address 127.5.232.190:40541)
0929 16:29:52.485283 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:52.485289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:52.485306 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:29:52.485513 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:52.485518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:52.485592 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:52.485692 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:52.486383 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.487117 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.488348 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.488773 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.488904 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:52.493414 (+ 4510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:39023: BlockingRecv error: recv got EOF from 127.5.232.131:39023 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 29 16:29:52 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:52 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:52.501894 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:52.494416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:54177)
0929 16:29:52.494552 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:52.494569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:52.494653 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:52.494942 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:52.494947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:52.495276 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:52.495478 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:52.495497 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.496554 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.496558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:52.497543 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:52.497553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.498197 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.498201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:52.498324 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:52.498800 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:52.498817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:52.501719 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 16:29:52.502144 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:52.494488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:54177 (local address 127.5.232.190:40541)
0929 16:29:52.494648 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:52.494653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:52.494669 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:29:52.494720 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:52.494724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:52.494797 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:52.494874 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:52.495608 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.496429 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.497659 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.498075 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.498167 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:52.501901 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:54177: BlockingRecv error: recv got EOF from 127.5.232.131:54177 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 16:29:52.502187 15273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 16:29:52 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:52 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:52.832290 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:52.823691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:55581)
0929 16:29:52.824206 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:52.824222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:52.824335 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:52.824623 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:52.824626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:52.825060 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:52.825211 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:52.825219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.826176 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.826180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:52.827115 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:52.827126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.827764 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.827768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:52.827879 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:52.828610 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:52.828627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:52.831928 (+ 3301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":304,"threads_started":1}
W20260929 16:29:52.832398 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:52.823812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:55581 (local address 127.5.232.190:40541)
0929 16:29:52.824081 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:52.824087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:52.824303 (+ 216us) server_negotiation.cc:407] Connection header received
0929 16:29:52.824431 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:52.824434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:52.824504 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:52.824585 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:52.825331 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.826049 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.827231 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:52.827616 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:52.827704 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:52.832153 (+ 4449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:55581: BlockingRecv error: recv got EOF from 127.5.232.130:55581 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 29 16:29:53 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:53 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:53.211150 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:53.201749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:57401)
0929 16:29:53.202324 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:53.202342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:53.202516 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:53.202870 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:53.202875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:53.203316 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:53.203498 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:53.203507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.204510 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.204515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:53.205528 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:53.205540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.206300 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.206305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:53.206530 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:53.207170 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:53.207188 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:53.210758 (+ 3570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":363,"threads_started":1}
W20260929 16:29:53.211202 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:53.201836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:57401 (local address 127.5.232.190:40541)
0929 16:29:53.202089 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:53.202102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:53.202452 (+ 350us) server_negotiation.cc:407] Connection header received
0929 16:29:53.202647 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:53.202654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:53.202723 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:53.202836 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:53.203649 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.204356 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.205668 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.206118 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.206258 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:53.210957 (+ 4699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:57401: BlockingRecv error: recv got EOF from 127.5.232.129:57401 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 29 16:29:53 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:53 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:53.511765 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:53.502810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:54645)
0929 16:29:53.503359 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:53.503374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:53.503545 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:53.503882 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:53.503887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:53.504323 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:53.504461 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:53.504469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.505578 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.505586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:53.506588 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:53.506600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.507333 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.507340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:53.507516 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:53.508058 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:53.508075 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:53.511403 (+ 3328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":324,"threads_started":1}
W20260929 16:29:53.511830 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:53.502904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:54645 (local address 127.5.232.190:40541)
0929 16:29:53.503155 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:53.503163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:53.503466 (+ 303us) server_negotiation.cc:407] Connection header received
0929 16:29:53.503672 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:53.503677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:53.503748 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:53.503854 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:53.504617 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.505441 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.506732 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.507152 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.507296 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:53.511601 (+ 4305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:54645: BlockingRecv error: recv got EOF from 127.5.232.131:54645 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 29 16:29:53 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:53 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:53.842326 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:53.833344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:36295)
0929 16:29:53.833919 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:53.833935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:53.834060 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:53.834416 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:53.834419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:53.834818 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:53.835002 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:53.835010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.836174 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.836178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:53.837279 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:53.837292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.838049 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.838053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:53.838190 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:53.838752 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:53.838769 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:53.842086 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":308,"threads_started":1}
W20260929 16:29:53.842504 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:53.833589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:36295 (local address 127.5.232.190:40541)
0929 16:29:53.833790 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:53.833794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:53.834015 (+ 221us) server_negotiation.cc:407] Connection header received
0929 16:29:53.834168 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:53.834172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:53.834233 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:53.834317 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:53.835143 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.835986 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.837490 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:53.837915 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:53.838014 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:53.842279 (+ 4265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:36295: BlockingRecv error: recv got EOF from 127.5.232.130:36295 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 16:29:54 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:54 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:54.221144 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:54.212164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:51451)
0929 16:29:54.212691 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:54.212705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:54.212859 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:54.213221 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:54.213225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:54.213630 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:54.213785 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:54.213794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.214812 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.214817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:54.215719 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:54.215730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.216426 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.216432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:54.216601 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:54.217273 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:54.217290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:54.220921 (+ 3631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":281,"threads_started":1}
W20260929 16:29:54.221334 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:54.212326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:51451 (local address 127.5.232.190:40541)
0929 16:29:54.212581 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:54.212589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:54.212789 (+ 200us) server_negotiation.cc:407] Connection header received
0929 16:29:54.212982 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:54.212987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:54.213057 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:54.213198 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:54.213930 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.214667 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.215860 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.216283 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.216413 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:54.221119 (+ 4706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:51451: BlockingRecv error: recv got EOF from 127.5.232.129:51451 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 29 16:29:54 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:54 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:54.521886 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:54.512884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:35381)
0929 16:29:54.513345 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:54.513362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:54.513510 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:54.513801 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:54.513804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:54.514171 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:54.514345 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:54.514354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.515442 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.515446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:54.516354 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:54.516366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.517112 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.517118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:54.517294 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:54.518059 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:54.518086 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:54.521639 (+ 3553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":270,"threads_started":1}
W20260929 16:29:54.522094 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:54.512962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:35381 (local address 127.5.232.190:40541)
0929 16:29:54.513214 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:54.513221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:54.513480 (+ 259us) server_negotiation.cc:407] Connection header received
0929 16:29:54.513603 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:54.513607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:54.513671 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:54.513760 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:54.514501 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.515259 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.516531 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.516965 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.517071 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:54.521845 (+ 4774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:35381: BlockingRecv error: recv got EOF from 127.5.232.131:35381 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 16:29:54 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:54 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:54.851648 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:54.843217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:32877)
0929 16:29:54.843682 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:54.843698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:54.843812 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:54.844087 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:54.844090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:54.844430 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:54.844588 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:54.844596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.845576 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.845579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:54.846468 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:54.846478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.847090 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.847094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:54.847217 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:54.847775 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:54.847792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:54.851305 (+ 3513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":266,"threads_started":1}
W20260929 16:29:54.851759 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:54.843432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:32877 (local address 127.5.232.190:40541)
0929 16:29:54.843677 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:54.843683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:54.843776 (+ 93us) server_negotiation.cc:407] Connection header received
0929 16:29:54.843904 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:54.843907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:54.843967 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:54.844046 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:54.844711 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.845452 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.846579 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:54.846970 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:54.847062 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:54.851496 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:32877: BlockingRecv error: recv got EOF from 127.5.232.130:32877 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 16:29:55 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:55 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:55.230728 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:55.222299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:57013)
0929 16:29:55.222956 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:55.222979 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:55.223112 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:55.223406 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:55.223409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:55.223765 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:55.223919 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:55.223926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.224965 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.224969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:55.226031 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:55.226043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.226656 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.226660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:55.226775 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:55.227341 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:55.227357 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:55.230483 (+ 3126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":422,"threads_started":1}
W20260929 16:29:55.230892 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:55.222369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:57013 (local address 127.5.232.190:40541)
0929 16:29:55.222575 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:55.222584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:55.223146 (+ 562us) server_negotiation.cc:407] Connection header received
0929 16:29:55.223216 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:55.223220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:55.223278 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:55.223358 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:55.224099 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.224842 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.226159 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.226539 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.226633 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:55.230672 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:57013: BlockingRecv error: recv got EOF from 127.5.232.129:57013 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 16:29:55 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:55 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:55.532236 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:55.522954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:40353)
0929 16:29:55.523494 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:55.523510 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:55.523621 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:55.523950 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:55.523953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:55.524311 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:55.524504 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:55.524517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.525846 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.525852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:55.526763 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:55.526775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.527456 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.527463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:55.527655 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:55.528342 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:55.528366 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:55.532022 (+ 3656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":310,"threads_started":1}
W20260929 16:29:55.532424 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:55.523145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:40353 (local address 127.5.232.190:40541)
0929 16:29:55.523319 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:55.523324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:55.523582 (+ 258us) server_negotiation.cc:407] Connection header received
0929 16:29:55.523754 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:55.523758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:55.523820 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:55.523909 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:55.524668 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.525698 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.526912 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.527302 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.527432 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:55.532203 (+ 4771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:40353: BlockingRecv error: recv got EOF from 127.5.232.131:40353 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 16:29:55 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:55 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:55.862020 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:55.852552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:54039)
0929 16:29:55.853109 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:55.853127 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:55.853278 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:55.853737 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:55.853743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:55.854232 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:55.854484 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:55.854496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.855625 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.855630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:55.856625 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:55.856636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.857287 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.857291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:55.857492 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:55.858165 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:55.858184 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:55.861687 (+ 3503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":320,"threads_started":1}
W20260929 16:29:55.862202 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:55.852669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:54039 (local address 127.5.232.190:40541)
0929 16:29:55.852860 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:55.852864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:55.853232 (+ 368us) server_negotiation.cc:407] Connection header received
0929 16:29:55.853449 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:55.853456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:55.853551 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:55.853666 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:55.854625 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.855487 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.856766 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:55.857161 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:55.857255 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:55.861892 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:54039: BlockingRecv error: recv got EOF from 127.5.232.130:54039 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 16:29:56 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:56 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:56.240502 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:56.231684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:48465)
0929 16:29:56.232237 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:56.232252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:56.232411 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:56.232756 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:56.232762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:56.233212 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:56.233356 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:56.233365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.234437 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.234444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:56.235361 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:56.235371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.236086 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.236092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:56.236274 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:56.236837 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:56.236855 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:56.240175 (+ 3320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":311,"threads_started":1}
W20260929 16:29:56.240607 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:56.231822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:48465 (local address 127.5.232.190:40541)
0929 16:29:56.232061 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:56.232069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:56.232341 (+ 272us) server_negotiation.cc:407] Connection header received
0929 16:29:56.232557 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:56.232560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:56.232619 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:56.232724 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:56.233541 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.234305 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.235503 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.235926 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.236089 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:56.240358 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:48465: BlockingRecv error: recv got EOF from 127.5.232.129:48465 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 16:29:56 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:56 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:56.542375 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:56.533321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:42141)
0929 16:29:56.533902 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:56.533919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:56.534037 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:56.534401 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:56.534404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:56.534805 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:56.534961 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:56.534969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.535943 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.535947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:56.536957 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:56.536969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.537677 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.537682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:56.537807 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:56.538580 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:56.538598 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:56.542126 (+ 3528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":353,"threads_started":1}
W20260929 16:29:56.542528 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:56.533438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:42141 (local address 127.5.232.190:40541)
0929 16:29:56.533612 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:56.533616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:56.534000 (+ 384us) server_negotiation.cc:407] Connection header received
0929 16:29:56.534157 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:56.534161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:56.534230 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:56.534315 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:56.535099 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.535810 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.537096 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.537526 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.537677 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:56.542310 (+ 4633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:42141: BlockingRecv error: recv got EOF from 127.5.232.131:42141 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 16:29:56 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:56 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:56.876047 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:56.863086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:39719)
0929 16:29:56.863609 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:56.863626 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:56.863737 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:56.864016 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:56.864019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:56.864415 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:56.864568 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:56.864576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.865581 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.865585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:56.866494 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:56.866504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.867436 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.867440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:56.867571 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:56.872416 (+ 4845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:56.872436 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:56.875822 (+ 3386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":284,"threads_started":1}
W20260929 16:29:56.876300 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:56.863305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:39719 (local address 127.5.232.190:40541)
0929 16:29:56.863494 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:56.863498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:56.863703 (+ 205us) server_negotiation.cc:407] Connection header received
0929 16:29:56.863836 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:56.863840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:56.863896 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:56.863983 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:56.864688 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.865427 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.866639 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:56.867299 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:56.867423 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:56.876035 (+ 8612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:39719: BlockingRecv error: recv got EOF from 127.5.232.130:39719 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:29:57 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:57 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:57.250356 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:57.241589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:38527)
0929 16:29:57.242039 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:57.242054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:57.242210 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:57.242534 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:57.242538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:57.242950 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:57.243113 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:57.243122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.244106 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.244113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:57.245115 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:57.245132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.245926 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.245932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:57.246106 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:57.246724 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:57.246742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:57.250025 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":231,"threads_started":1}
W20260929 16:29:57.250450 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:57.241643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:38527 (local address 127.5.232.190:40541)
0929 16:29:57.241866 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:57.241875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:57.242144 (+ 269us) server_negotiation.cc:407] Connection header received
0929 16:29:57.242343 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:57.242347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:57.242404 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:57.242508 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:57.243266 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.243969 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.245245 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.245768 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.245904 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:57.250213 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:38527: BlockingRecv error: recv got EOF from 127.5.232.129:38527 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 16:29:57 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:57 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:57.553428 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:57.543378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:58441)
0929 16:29:57.543942 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:57.543958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:57.544078 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:57.544374 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:57.544377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:57.544777 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:57.544924 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:57.544932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.546122 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.546128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:57.547070 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:57.547081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.547981 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.547987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:57.548185 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:57.548929 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:57.548954 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:57.553090 (+ 4136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":329,"threads_started":1}
W20260929 16:29:57.553596 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:57.543516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:58441 (local address 127.5.232.190:40541)
0929 16:29:57.543709 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:57.543715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:57.544075 (+ 360us) server_negotiation.cc:407] Connection header received
0929 16:29:57.544169 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:57.544173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:57.544236 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:57.544331 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:57.545104 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.545968 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.547238 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.547821 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.547968 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:57.553278 (+ 5310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:58441: BlockingRecv error: recv got EOF from 127.5.232.131:58441 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 16:29:57 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:57 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:57.885507 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:57.877042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:40249)
0929 16:29:57.877555 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:57.877572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:57.877688 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:57.877961 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:57.877964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:57.878345 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:57.878516 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:57.878524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.879499 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.879503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:57.880402 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:57.880413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.881041 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.881045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:57.881177 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:57.881786 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:57.881803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:57.885213 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":267,"threads_started":1}
W20260929 16:29:57.885807 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:57.877087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:40249 (local address 127.5.232.190:40541)
0929 16:29:57.877290 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:57.877296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:57.877647 (+ 351us) server_negotiation.cc:407] Connection header received
0929 16:29:57.877774 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:57.877778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:57.877839 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:57.877920 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:57.878648 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.879383 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.880522 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:57.880911 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:57.881002 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:57.885540 (+ 4538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:40249: BlockingRecv error: recv got EOF from 127.5.232.130:40249 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:29:58 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:58 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:58.260715 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:58.251483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:37479)
0929 16:29:58.252134 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:58.252150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:58.252276 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:58.252599 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:58.252602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:58.253026 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:58.253266 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:58.253277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.254702 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.254706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:58.255647 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:58.255658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.256323 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.256327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:58.256439 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:58.257016 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:58.257033 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:58.260471 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":324,"threads_started":1}
W20260929 16:29:58.260914 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:58.251601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:37479 (local address 127.5.232.190:40541)
0929 16:29:58.251752 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:58.251757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:58.252232 (+ 475us) server_negotiation.cc:407] Connection header received
0929 16:29:58.252378 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:58.252382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:58.252445 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:58.252587 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:58.253443 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.254531 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.255778 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.256190 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.256284 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:58.260684 (+ 4400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:37479: BlockingRecv error: recv got EOF from 127.5.232.129:37479 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 16:29:58 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:58 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:58.563494 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:58.554673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:35587)
0929 16:29:58.555210 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:58.555230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:58.555382 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:58.555793 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:58.555798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:58.556317 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:58.556501 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:58.556509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.557537 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.557541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:58.558444 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:58.558454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.559158 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.559162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:58.559295 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:58.559848 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:58.559868 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:58.563274 (+ 3406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
W20260929 16:29:58.563798 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:58.554769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:35587 (local address 127.5.232.190:40541)
0929 16:29:58.554936 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:58.554942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:58.555333 (+ 391us) server_negotiation.cc:407] Connection header received
0929 16:29:58.555534 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:58.555539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:58.555638 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:58.555736 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:58.556638 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.557329 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.558607 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.559013 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.559118 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:58.563499 (+ 4381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:35587: BlockingRecv error: recv got EOF from 127.5.232.131:35587 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 29 16:29:58 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:58 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:58.895651 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:58.886524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:37999)
0929 16:29:58.887110 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:58.887127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:58.887238 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:58.887500 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:58.887503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:58.887909 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:58.888087 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:58.888096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.889021 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.889024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:58.890033 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:58.890044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.890768 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.890773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:58.890960 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:58.891659 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:58.891676 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:58.895409 (+ 3733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":273,"threads_started":1}
W20260929 16:29:58.895848 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:58.886695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:37999 (local address 127.5.232.190:40541)
0929 16:29:58.886932 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:58.886939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:58.887200 (+ 261us) server_negotiation.cc:407] Connection header received
0929 16:29:58.887320 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:58.887324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:58.887383 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:58.887462 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:58.888205 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.888905 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.890158 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:58.890631 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:58.890724 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:58.895621 (+ 4897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:37999: BlockingRecv error: recv got EOF from 127.5.232.130:37999 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:29:59 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:59 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:59.270241 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:59.261652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:44809)
0929 16:29:59.262269 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:59.262285 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:59.262395 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:59.262669 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:59.262672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:59.263126 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:59.263291 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:59.263300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.264233 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.264236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:59.265156 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:59.265166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.266007 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.266012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:59.266126 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:59.266692 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:59.266709 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:59.269949 (+ 3240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":340,"threads_started":1}
W20260929 16:29:59.270578 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:59.261769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:44809 (local address 127.5.232.190:40541)
0929 16:29:59.261981 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:59.261987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:59.262362 (+ 375us) server_negotiation.cc:407] Connection header received
0929 16:29:59.262487 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:59.262491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:59.262549 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:59.262632 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:59.263420 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.264105 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.265304 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.265861 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.265960 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:59.270326 (+ 4366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:44809: BlockingRecv error: recv got EOF from 127.5.232.129:44809 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 16:29:59 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:59 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:59.572919 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:59.564494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:59063)
0929 16:29:59.565028 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:59.565046 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:59.565167 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:59.565455 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:59.565459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:59.565858 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:59.566024 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:59.566032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.567017 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.567020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:59.567946 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:59.567956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.568601 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.568605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:59.568713 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:59.569419 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:59.569451 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:59.572688 (+ 3237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":250,"threads_started":1}
W20260929 16:29:59.573215 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:59.564578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:59063 (local address 127.5.232.190:40541)
0929 16:29:59.564777 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:59.564801 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:59.565138 (+ 337us) server_negotiation.cc:407] Connection header received
0929 16:29:59.565258 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:59.565262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:59.565323 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:59.565434 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:59.566185 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.566892 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.568064 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.568476 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.568568 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:59.572913 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:59063: BlockingRecv error: recv got EOF from 127.5.232.131:59063 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 16:29:59 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:29:59 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:29:59.905422 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:59.896608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:42687)
0929 16:29:59.897122 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:59.897138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:59.897252 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:59.897551 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:59.897554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:59.897936 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:59.898100 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:59.898108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.899260 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.899264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:59.900211 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:59.900223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.900869 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.900873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:59.900984 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:59.901737 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:59.901756 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:59.905072 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":269,"threads_started":1}
W20260929 16:29:59.905812 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:59.896669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:42687 (local address 127.5.232.190:40541)
0929 16:29:59.896855 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:59.896859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:59.897212 (+ 353us) server_negotiation.cc:407] Connection header received
0929 16:29:59.897345 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:59.897349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:59.897428 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:59.897516 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:59.898220 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.899137 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.900330 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:59.900722 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:59.900812 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:59.905507 (+ 4695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:42687: BlockingRecv error: recv got EOF from 127.5.232.130:42687 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 16:30:00 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:00 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:00.280817 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:00.271420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:55363)
0929 16:30:00.271978 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:00.271994 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:00.272152 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:00.272808 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:00.272815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:00.273434 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:00.273587 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:00.273596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.274607 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.274610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:00.275536 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:00.275546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.276243 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.276247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:00.276371 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:00.276942 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:00.276959 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:00.280539 (+ 3580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":326,"threads_started":1}
W20260929 16:30:00.280941 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:00.271507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:55363 (local address 127.5.232.190:40541)
0929 16:30:00.271701 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:00.271706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:00.272105 (+ 399us) server_negotiation.cc:407] Connection header received
0929 16:30:00.272324 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:00.272332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:00.272411 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:00.272500 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:00.273733 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.274487 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.275664 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.276105 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.276202 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:00.280718 (+ 4516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:55363: BlockingRecv error: recv got EOF from 127.5.232.129:55363 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 16:30:00 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:00 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:00.583535 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:00.574127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:48087)
0929 16:30:00.574714 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:00.574730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:00.574855 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:00.575152 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:00.575155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:00.575530 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:00.575694 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:00.575703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.576726 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.576730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:00.577833 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:00.577851 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.578760 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.578767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:00.578922 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:00.579692 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:00.579721 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:00.583294 (+ 3573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":324,"threads_started":1}
W20260929 16:30:00.583732 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:00.574235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:48087 (local address 127.5.232.190:40541)
0929 16:30:00.574412 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:00.574416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:00.574807 (+ 391us) server_negotiation.cc:407] Connection header received
0929 16:30:00.574956 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:00.574960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:00.575021 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:00.575108 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:00.575872 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.576590 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.578019 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.578571 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.578714 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:00.583502 (+ 4788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:48087: BlockingRecv error: recv got EOF from 127.5.232.131:48087 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:30:00 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:00 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:00.915676 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:00.906709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:56017)
0929 16:30:00.907212 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:00.907227 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:00.907384 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:00.907783 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:00.907787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:00.908229 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:00.908364 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:00.908372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.909723 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.909727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:00.910730 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:00.910741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.911399 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.911402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:00.911521 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:00.912064 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:00.912080 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:00.915415 (+ 3335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":257,"threads_started":1}
W20260929 16:30:00.915854 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:00.906759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:56017 (local address 127.5.232.190:40541)
0929 16:30:00.906989 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:00.906997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:00.907318 (+ 321us) server_negotiation.cc:407] Connection header received
0929 16:30:00.907540 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:00.907545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:00.907648 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:00.907758 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:00.908527 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.909556 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.910852 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:00.911274 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:00.911369 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:00.915600 (+ 4231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:56017: BlockingRecv error: recv got EOF from 127.5.232.130:56017 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 16:30:01 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:01 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:01.291704 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:01.282048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:53581)
0929 16:30:01.282606 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:01.282622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:01.282738 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:01.283045 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:01.283049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:01.283442 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:01.283604 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:01.283614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.284607 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.284611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:01.285712 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:01.285726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.286702 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.286707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:01.286911 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:01.287693 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:01.287713 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:01.291356 (+ 3643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":321,"threads_started":1}
W20260929 16:30:01.291807 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:01.282212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:53581 (local address 127.5.232.190:40541)
0929 16:30:01.282379 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:01.282383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:01.282698 (+ 315us) server_negotiation.cc:407] Connection header received
0929 16:30:01.282848 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:01.282851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:01.282913 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:01.283003 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:01.283799 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.284476 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.285885 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.286541 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.286672 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:01.291573 (+ 4901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:53581: BlockingRecv error: recv got EOF from 127.5.232.129:53581 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 16:30:01 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:01 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:01.593674 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:01.584538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:51655)
0929 16:30:01.585030 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:01.585045 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:01.585202 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:01.585578 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:01.585584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:01.586074 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:01.586211 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:01.586219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.587325 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.587332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:01.588265 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:01.588276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.588959 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.588965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:01.589145 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:01.589687 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:01.589703 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:01.593324 (+ 3621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":252,"threads_started":1}
W20260929 16:30:01.593871 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:01.584600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:51655 (local address 127.5.232.190:40541)
0929 16:30:01.584869 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:01.584878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:01.585143 (+ 265us) server_negotiation.cc:407] Connection header received
0929 16:30:01.585339 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:01.585345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:01.585440 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:01.585554 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:01.586355 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.587193 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.588411 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.588811 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.588944 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:01.593562 (+ 4618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:51655: BlockingRecv error: recv got EOF from 127.5.232.131:51655 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 16:30:01 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:01 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:01.929255 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:01.916890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:47803)
0929 16:30:01.917397 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:01.917413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:01.917532 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:01.917831 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:01.917834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:01.918298 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:01.918460 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:01.918469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.919454 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.919457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:01.920449 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:01.920461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.921162 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.921166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:01.921296 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:01.925091 (+ 3795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:01.925112 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:01.928903 (+ 3791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":328,"threads_started":1}
W20260929 16:30:01.929422 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:01.916960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:47803 (local address 127.5.232.190:40541)
0929 16:30:01.917217 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:01.917221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:01.917477 (+ 256us) server_negotiation.cc:407] Connection header received
0929 16:30:01.917625 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:01.917629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:01.917702 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:01.917795 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:01.918602 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.919336 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.920581 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:01.921025 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:01.921131 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:01.929123 (+ 7992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:47803: BlockingRecv error: recv got EOF from 127.5.232.130:47803 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 16:30:02 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:02 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:02.305461 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:02.292670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:60821)
0929 16:30:02.293190 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:02.293209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:02.293397 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:02.293751 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:02.293757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:02.294330 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:02.294580 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:02.294591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.296808 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.296814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:02.298094 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:02.298107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.299312 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.299329 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:02.299548 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:02.300279 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:02.300468 (+ 189us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:02.305092 (+ 4624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":279,"threads_started":1}
W20260929 16:30:02.305737 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:02.292881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:60821 (local address 127.5.232.190:40541)
0929 16:30:02.293103 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:02.293110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:02.293299 (+ 189us) server_negotiation.cc:407] Connection header received
0929 16:30:02.293529 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:02.293534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:02.293609 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:02.293715 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:02.295370 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.296669 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.298247 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.298880 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.299177 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:02.305461 (+ 6284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:60821: BlockingRecv error: recv got EOF from 127.5.232.129:60821 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 16:30:02 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:02 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:02.603857 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:02.594817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:52655)
0929 16:30:02.595362 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:02.595378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:02.595543 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:02.595903 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:02.595907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:02.596315 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:02.596470 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:02.596479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.597530 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.597537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:02.598527 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:02.598539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.599280 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.599286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:02.599447 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:02.600042 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:02.600059 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:02.603509 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
W20260929 16:30:02.603950 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:02.595046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:52655 (local address 127.5.232.190:40541)
0929 16:30:02.595247 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:02.595256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:02.595473 (+ 217us) server_negotiation.cc:407] Connection header received
0929 16:30:02.595688 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:02.595698 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:02.595760 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:02.595871 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:02.596629 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.597345 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.598690 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.599109 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.599249 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:02.603715 (+ 4466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:52655: BlockingRecv error: recv got EOF from 127.5.232.131:52655 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 16:30:02 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:02 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:02.939208 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:02.930233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:57201)
0929 16:30:02.930748 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:02.930765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:02.930879 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:02.931187 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:02.931190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:02.931605 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:02.931786 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:02.931795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.932805 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.932809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:02.933769 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:02.933781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.934468 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.934472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:02.934599 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:02.935191 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:02.935224 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:02.938869 (+ 3645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":300,"threads_started":1}
W20260929 16:30:02.939368 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:02.930354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:57201 (local address 127.5.232.190:40541)
0929 16:30:02.930524 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:02.930529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:02.930842 (+ 313us) server_negotiation.cc:407] Connection header received
0929 16:30:02.930978 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:02.930982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:02.931056 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:02.931141 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:02.931924 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.932671 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.933895 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:02.934315 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:02.934408 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:02.939076 (+ 4668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:57201: BlockingRecv error: recv got EOF from 127.5.232.130:57201 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 16:30:03 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:03 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:03.318012 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:03.309269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:36811)
0929 16:30:03.309798 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:03.309822 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:03.309934 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:03.310224 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:03.310227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:03.310606 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:03.310773 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:03.310782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.311750 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.311754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:03.312687 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:03.312698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.313433 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.313438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:03.313577 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:03.314160 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:03.314178 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:03.317677 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":302,"threads_started":1}
W20260929 16:30:03.318216 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:03.309283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:36811 (local address 127.5.232.190:40541)
0929 16:30:03.309523 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:03.309533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:03.309905 (+ 372us) server_negotiation.cc:407] Connection header received
0929 16:30:03.310033 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:03.310036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:03.310097 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:03.310180 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:03.310914 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.311610 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.312824 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.313240 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.313338 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:03.317975 (+ 4637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:36811: BlockingRecv error: recv got EOF from 127.5.232.129:36811 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 16:30:03 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:03 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:03.614220 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:03.604898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:32989)
0929 16:30:03.605630 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:03.605646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:03.605809 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:03.606156 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:03.606159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:03.606600 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:03.606783 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:03.606792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.607853 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.607861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:03.608797 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:03.608807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.609601 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.609607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:03.609790 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:03.610435 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:03.610453 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:03.613855 (+ 3402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":516,"threads_started":1}
W20260929 16:30:03.614318 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:03.604965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:32989 (local address 127.5.232.190:40541)
0929 16:30:03.605261 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:03.605270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:03.605756 (+ 486us) server_negotiation.cc:407] Connection header received
0929 16:30:03.605936 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:03.605941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:03.606027 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:03.606111 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:03.606950 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.607714 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.608942 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.609443 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.609583 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:03.614063 (+ 4480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:32989: BlockingRecv error: recv got EOF from 127.5.232.131:32989 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 16:30:03 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:03 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:03.950176 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:03.940325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:43567)
0929 16:30:03.940902 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:03.940919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:03.941084 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:03.941494 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:03.941499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:03.941946 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:03.942109 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:03.942117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.943127 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.943132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:03.944058 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:03.944069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.944867 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.944873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:03.945049 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:03.945871 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:03.945901 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:03.949803 (+ 3902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":309,"threads_started":1}
W20260929 16:30:03.950322 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:03.940427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:43567 (local address 127.5.232.190:40541)
0929 16:30:03.940688 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:03.940696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:03.941013 (+ 317us) server_negotiation.cc:407] Connection header received
0929 16:30:03.941214 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:03.941221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:03.941320 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:03.941462 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:03.942263 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.942978 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.944210 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.944680 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.944830 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:03.950060 (+ 5230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:43567: BlockingRecv error: recv got EOF from 127.5.232.130:43567 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 16:30:04 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:04 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:04.327998 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.319074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:39165)
0929 16:30:04.319579 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.319597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.319714 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.320014 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.320017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.320432 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:04.320574 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.320582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.321682 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.321689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.322626 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.322638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.323383 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.323389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.323579 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.324174 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:04.324192 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:04.327770 (+ 3578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":308,"threads_started":1}
W20260929 16:30:04.328315 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:04.319122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:39165 (local address 127.5.232.190:40541)
0929 16:30:04.319351 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.319360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.319676 (+ 316us) server_negotiation.cc:407] Connection header received
0929 16:30:04.319813 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.319817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.319887 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.319968 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:04.320700 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.321534 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.322787 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.323221 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.323364 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.328051 (+ 4687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:39165: BlockingRecv error: recv got EOF from 127.5.232.129:39165 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 16:30:04 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:04 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:04.623924 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.615137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:47061)
0929 16:30:04.615736 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.615753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.615868 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.616184 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.616188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.616586 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:04.616765 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.616775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.617966 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.617970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.618925 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.618938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.619627 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.619632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.619773 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.620367 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:04.620384 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:04.623656 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":334,"threads_started":1}
W20260929 16:30:04.624083 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:04.615222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:47061 (local address 127.5.232.190:40541)
0929 16:30:04.615397 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.615401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.615833 (+ 432us) server_negotiation.cc:407] Connection header received
0929 16:30:04.615963 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.615966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.616025 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.616110 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:04.616893 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.617827 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.619048 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.619469 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.619567 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.623879 (+ 4312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:47061: BlockingRecv error: recv got EOF from 127.5.232.131:47061 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 16:30:04 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:04 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:04.960220 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.951300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:46191)
0929 16:30:04.951883 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.951899 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.952013 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.952320 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.952323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.952706 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:04.952853 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.952862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.953887 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.953890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.954804 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.954815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.955577 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.955582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.955718 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.956399 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:04.956417 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:04.959993 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":374,"threads_started":1}
W20260929 16:30:04.960433 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:04.951464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:46191 (local address 127.5.232.190:40541)
0929 16:30:04.951674 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.951679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.951980 (+ 301us) server_negotiation.cc:407] Connection header received
0929 16:30:04.952110 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.952114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.952188 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.952277 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:04.952980 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.953752 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.954938 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.955421 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.955522 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.960192 (+ 4670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:46191: BlockingRecv error: recv got EOF from 127.5.232.130:46191 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 16:30:05 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:05 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:05.339203 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.329727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:35181)
0929 16:30:05.330434 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.330451 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.330567 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.330955 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.330960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.331365 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:05.331552 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.331561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.332592 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.332599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.333571 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.333584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.334316 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.334322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.334508 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.335147 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:05.335165 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:05.338825 (+ 3660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":271,"threads_started":1}
W20260929 16:30:05.339341 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:05.329797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:35181 (local address 127.5.232.190:40541)
0929 16:30:05.330068 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.330075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.330561 (+ 486us) server_negotiation.cc:407] Connection header received
0929 16:30:05.330722 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.330728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.330793 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.330924 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:05.331729 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.332433 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.333737 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.334151 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.334302 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.339079 (+ 4777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:35181: BlockingRecv error: recv got EOF from 127.5.232.129:35181 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 16:30:05 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:05 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.131@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:05.634542 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.624984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.131:36061)
0929 16:30:05.625538 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.625557 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.625681 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.626065 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.626069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.626468 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:05.626626 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.626635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.627814 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.627820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.628733 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.628746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.629578 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.629585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.629767 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.630459 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:05.630481 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:05.634278 (+ 3797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":259,"threads_started":1}
W20260929 16:30:05.634816 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:05.625252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:36061 (local address 127.5.232.190:40541)
0929 16:30:05.625577 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.625582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.625620 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:30:05.625796 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.625800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.625872 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.625965 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:05.626823 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.627643 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.628904 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.629337 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.629552 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.634528 (+ 4976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.131:36061: BlockingRecv error: recv got EOF from 127.5.232.131:36061 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Sep 29 16:30:05 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:05 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.130@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:05.970916 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.961694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.130:46061)
0929 16:30:05.962202 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.962221 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.962368 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.962750 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.962755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.963156 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:05.963385 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.963400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.964497 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.964505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.965498 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.965511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.966253 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.966259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.966448 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.967135 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:05.967153 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:05.970647 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":360,"threads_started":1}
W20260929 16:30:05.971117 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:05.961847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:46061 (local address 127.5.232.190:40541)
0929 16:30:05.962051 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.962058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.962296 (+ 238us) server_negotiation.cc:407] Connection header received
0929 16:30:05.962521 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.962528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.962612 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.962720 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:05.963552 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.964347 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.965659 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.966089 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.966224 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.970888 (+ 4664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.130:46061: BlockingRecv error: recv got EOF from 127.5.232.130:46061 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 16:30:06 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:30:06 dist-test-slave-dssx krb5kdc[14272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.232.129@KRBTEST.COM for oryx/127.5.232.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:30:06.350608 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:06.340360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40541 (local address 127.5.232.129:42487)
0929 16:30:06.340924 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:06.340940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:06.341105 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:06.341442 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:06.341445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:06.341843 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:06.342010 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:06.342018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.343367 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.343371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:06.344623 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:06.344635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.345475 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.345485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:06.345624 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:06.346274 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:06.346292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:06.350288 (+ 3996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.232.190:40541: Server oryx/127.5.232.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":325,"threads_started":1}
W20260929 16:30:06.350796 15008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:06.340505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:42487 (local address 127.5.232.190:40541)
0929 16:30:06.340696 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:06.340701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:06.341092 (+ 391us) server_negotiation.cc:407] Connection header received
0929 16:30:06.341214 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:06.341218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:06.341298 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:06.341430 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:06.342141 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.343217 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.344759 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.345191 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.345302 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:06.350539 (+ 5237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.232.129:42487: BlockingRecv error: recv got EOF from 127.5.232.129:42487 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:30:06.354722 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 14859
I20260929 16:30:06.367352 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:06.355866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.5.232.129:35155)
0929 16:30:06.356505 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:06.356512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:06.356544 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:30:06.356612 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:06.356620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:06.356857 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:06.357038 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:06.358485 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.359556 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.360818 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.361480 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.361591 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:06.364139 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:06.364148 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:06.364163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:06.364203 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:06.366795 (+ 2592us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.232.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 16:30:06.367098 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39278: Request ticket server kudu/127.5.232.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":319,"threads_started":1}
W20260929 16:30:06.367545 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 14859: Not authorized: Client connection negotiation failed: client connection to 127.5.232.129:35155: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.232.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 16:30:06.367753 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 14859
I20260929 16:30:06.384737 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15010
I20260929 16:30:06.397863 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:06.385908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.5.232.130:44823)
0929 16:30:06.386489 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:06.386496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:06.386525 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:30:06.386588 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:06.386594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:06.386840 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:06.387092 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:06.388435 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.389644 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.390937 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.391598 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.391758 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:06.394536 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:06.394545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:06.394561 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:06.394601 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:06.397104 (+ 2503us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.232.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 16:30:06.397512 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33064: Request ticket server kudu/127.5.232.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":281,"threads_started":1}
W20260929 16:30:06.397871 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15010: Not authorized: Client connection negotiation failed: client connection to 127.5.232.130:44823: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.232.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 16:30:06.397974 15323 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33064: Request ticket server kudu/127.5.232.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 16:30:06.398030 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 15010
I20260929 16:30:06.411558 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15143
I20260929 16:30:06.424633 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:06.412807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.5.232.131:42415)
0929 16:30:06.413540 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:06.413549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:06.413581 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:30:06.413665 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:06.413673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:06.414023 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:06.414258 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:06.415582 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.416594 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.418038 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.418592 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.418698 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:06.421254 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:06.421263 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:06.421278 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:06.421319 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:06.424083 (+ 2764us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.232.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 16:30:06.424384 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34470: Request ticket server kudu/127.5.232.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":357,"threads_started":1}
W20260929 16:30:06.424767 15324 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34470: Request ticket server kudu/127.5.232.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 16:30:06.424821 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15143: Not authorized: Client connection negotiation failed: client connection to 127.5.232.131:42415: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.232.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 16:30:06.425024 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 15143
I20260929 16:30:06.442653 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 14790
I20260929 16:30:06.444061 14852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:06.561729 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 14790
2026-09-29T16:30:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21455 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T16:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:06Z Disabled control of system clock
I20260929 16:30:06.639470 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:33825
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:33825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:30:06.990615 15329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:06.991029 15329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:06.991187 15329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:06.998716 15329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:06.998847 15329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:06.998935 15329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:06.999034 15329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:06.999114 15329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:07.008673 15329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:33825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:33825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:07.010072 15329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:07.012028 15329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:07.024724 15337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:07.024683 15335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:07.024762 15334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:07.025072 15329 server_base.cc:1034] running on GCE node
I20260929 16:30:07.026180 15329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:07.027602 15329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:07.028824 15329 hybrid_clock.cc:648] HybridClock initialized: now 1790699407028776 us; error 49 us; skew 500 ppm
I20260929 16:30:07.029151 15329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:30:07.035336 15329 webserver.cc:460] Webserver started at http://127.5.232.190:36099/ using document root <none> and password file <none>
I20260929 16:30:07.036064 15329 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:07.036155 15329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:07.036442 15329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:07.039475 15329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "a53fcdff0e954e858b6e876ae3e97cf5"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-dssx"
server_key: "010a56a0a0143db4a5975feccb19908c"
server_key_iv: "04caf58287bc52dcdae5b4429e60d362"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.040256 15329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "a53fcdff0e954e858b6e876ae3e97cf5"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-dssx"
server_key: "010a56a0a0143db4a5975feccb19908c"
server_key_iv: "04caf58287bc52dcdae5b4429e60d362"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.045897 15329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 16:30:07.049510 15343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.050995 15329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:30:07.051203 15329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "a53fcdff0e954e858b6e876ae3e97cf5"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-dssx"
server_key: "010a56a0a0143db4a5975feccb19908c"
server_key_iv: "04caf58287bc52dcdae5b4429e60d362"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.051453 15329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:07.075131 15329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:07.076122 15329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:07.076452 15329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:07.096063 15329 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:33825
I20260929 16:30:07.096093 15394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:33825 every 8 connection(s)
I20260929 16:30:07.097573 15329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:07.101676 15395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:07.103207 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 15329
I20260929 16:30:07.103439 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:30:07.104076 6050 external_mini_cluster.cc:1428] Setting key 2b207c8a8a3e179e8fbd75c6e133baa6
I20260929 16:30:07.115902 15395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5: Bootstrap starting.
I20260929 16:30:07.120030 15395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:07.120934 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:07.106344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.5.232.190:33825)
0929 16:30:07.107606 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:07.107627 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:07.107684 (+ 57us) server_negotiation.cc:407] Connection header received
0929 16:30:07.110147 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:07.110178 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:07.110826 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:07.111306 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:07.111905 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.113986 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.115388 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.116471 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.116653 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:07.117851 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:07.117868 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:07.117893 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:07.118640 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:07.118764 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:07.118790 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:07.118923 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:07.119140 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 16:30:07.119487 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":439,"threads_started":1}
I20260929 16:30:07.121466 15395 log.cc:824] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:07.124912 15395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5: No bootstrap required, opened a new log
I20260929 16:30:07.133167 15395 raft_consensus.cc:348] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53fcdff0e954e858b6e876ae3e97cf5" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33825 } }
I20260929 16:30:07.133767 15395 raft_consensus.cc:374] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:07.133893 15395 raft_consensus.cc:729] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53fcdff0e954e858b6e876ae3e97cf5, State: Initialized, Role: FOLLOWER
I20260929 16:30:07.134627 15395 consensus_queue.cc:260] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53fcdff0e954e858b6e876ae3e97cf5" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33825 } }
I20260929 16:30:07.134894 15395 raft_consensus.cc:388] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:07.135066 15395 raft_consensus.cc:482] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:07.135250 15395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:07.138329 15395 raft_consensus.cc:504] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53fcdff0e954e858b6e876ae3e97cf5" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33825 } }
I20260929 16:30:07.138988 15395 leader_election.cc:302] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a53fcdff0e954e858b6e876ae3e97cf5; no voters:
I20260929 16:30:07.139611 15395 leader_election.cc:288] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:07.139796 15400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:07.140515 15400 raft_consensus.cc:686] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [term 1 LEADER]: Becoming Leader. State: Replica: a53fcdff0e954e858b6e876ae3e97cf5, State: Running, Role: LEADER
I20260929 16:30:07.141065 15400 consensus_queue.cc:237] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53fcdff0e954e858b6e876ae3e97cf5" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33825 } }
I20260929 16:30:07.142017 15395 sys_catalog.cc:564] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:07.144130 15402 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a53fcdff0e954e858b6e876ae3e97cf5. Latest consensus state: current_term: 1 leader_uuid: "a53fcdff0e954e858b6e876ae3e97cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53fcdff0e954e858b6e876ae3e97cf5" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33825 } } }
I20260929 16:30:07.144604 15402 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:07.144137 15401 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a53fcdff0e954e858b6e876ae3e97cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53fcdff0e954e858b6e876ae3e97cf5" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33825 } } }
I20260929 16:30:07.147063 15406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:07.147101 15401 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:07.152908 15406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:07.162626 15406 catalog_manager.cc:1347] Generated new cluster ID: 612390be0e69458fab43b844225cb40b
I20260929 16:30:07.162763 15406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:07.173499 15406 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:07.174901 15406 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:07.187399 15406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a53fcdff0e954e858b6e876ae3e97cf5: Generated new TSK 0
I20260929 16:30:07.188329 15406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:07.196730 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:33825
--builtin_ntp_servers=127.5.232.148:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 16:30:07.545600 15419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:07.545915 15419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:07.546078 15419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:07.553937 15419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:07.554049 15419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:07.554260 15419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:30:07.563930 15419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:33825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:07.565313 15419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:07.567239 15419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:07.581477 15424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:07.581454 15427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:07.581413 15425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:07.582999 15419 server_base.cc:1034] running on GCE node
I20260929 16:30:07.583750 15419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:07.584751 15419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:07.585943 15419 hybrid_clock.cc:648] HybridClock initialized: now 1790699407585878 us; error 57 us; skew 500 ppm
I20260929 16:30:07.586297 15419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:30:07.588762 15419 webserver.cc:460] Webserver started at http://127.5.232.129:37559/ using document root <none> and password file <none>
I20260929 16:30:07.589618 15419 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:07.589746 15419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:07.590092 15419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:07.593015 15419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "4fb39f6456bc4cf38d7242c039bedeba"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-dssx"
server_key: "fa532f7b0e4083087b038e213fb377b5"
server_key_iv: "4cedd55d64f4a165e2f1146d9cf18592"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.593914 15419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "4fb39f6456bc4cf38d7242c039bedeba"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-dssx"
server_key: "fa532f7b0e4083087b038e213fb377b5"
server_key_iv: "4cedd55d64f4a165e2f1146d9cf18592"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.599383 15419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:30:07.603624 15433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.605938 15419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:30:07.606189 15419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "4fb39f6456bc4cf38d7242c039bedeba"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-dssx"
server_key: "fa532f7b0e4083087b038e213fb377b5"
server_key_iv: "4cedd55d64f4a165e2f1146d9cf18592"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.606489 15419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:07.620530 15419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:07.621527 15419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:07.621838 15419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:07.622838 15419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:07.624475 15419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:07.624600 15419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.624734 15419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:07.624825 15419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.658927 15419 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:44567
I20260929 16:30:07.658972 15545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:44567 every 8 connection(s)
I20260929 16:30:07.660266 15419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:07.662523 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 15419
I20260929 16:30:07.662743 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:07.663568 6050 external_mini_cluster.cc:1428] Setting key d0790551246aa9225129a40b15995d9f
I20260929 16:30:07.666666 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:33825
--builtin_ntp_servers=127.5.232.148:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:30:07.674376 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:07.662991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:40867 (local address 127.5.232.190:33825)
0929 16:30:07.663466 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:07.663471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:07.665082 (+ 1611us) server_negotiation.cc:407] Connection header received
0929 16:30:07.667157 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:07.667165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:07.667256 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:07.667380 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:07.668488 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.669686 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.671275 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.671997 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.672145 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:07.673599 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:07.673606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:07.673610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:07.673639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:07.673680 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:07.673692 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:07.673761 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:07.674040 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 16:30:07.674119 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":268,"threads_started":1}
I20260929 16:30:07.675942 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:07.663774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:33825 (local address 127.5.232.129:40867)
0929 16:30:07.664901 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:07.664957 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:07.666824 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:07.667603 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:07.667612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:07.667709 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:07.668307 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:07.668324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.669834 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.669838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:07.671119 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:07.671130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.672208 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.672212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:07.672422 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:07.673202 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:07.673205 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:07.673246 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:07.673329 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:07.673789 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:07.673802 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:07.673811 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:30:07.674030 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 16:30:07.674508 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":329,"threads_started":1}
I20260929 16:30:07.677595 15546 heartbeater.cc:347] Connected to a master server at 127.5.232.190:33825
I20260929 16:30:07.677918 15546 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:07.678685 15546 heartbeater.cc:511] Master 127.5.232.190:33825 requested a full tablet report, sending...
I20260929 16:30:07.680775 15360 ts_manager.cc:194] Registered new tserver with Master: 4fb39f6456bc4cf38d7242c039bedeba (127.5.232.129:44567)
I20260929 16:30:07.682369 15360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.129:40867
W20260929 16:30:08.010264 15550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:08.010654 15550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:08.010818 15550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:08.018345 15550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:08.018510 15550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:08.018688 15550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:30:08.028204 15550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:33825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:08.029637 15550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:08.031414 15550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:08.045042 15555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:08.045136 15556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:08.045158 15558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:08.046036 15550 server_base.cc:1034] running on GCE node
I20260929 16:30:08.046674 15550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:08.047590 15550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:08.048791 15550 hybrid_clock.cc:648] HybridClock initialized: now 1790699408048718 us; error 66 us; skew 500 ppm
I20260929 16:30:08.049126 15550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:30:08.051569 15550 webserver.cc:460] Webserver started at http://127.5.232.130:43971/ using document root <none> and password file <none>
I20260929 16:30:08.052244 15550 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:08.052333 15550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:08.052608 15550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:08.055575 15550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "54a1953220c04b03a5dbde97fc72135e"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-dssx"
server_key: "93adb0bd950247538c614a5528f0558d"
server_key_iv: "4a607f954eea74530b9845baa9b41c6c"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.056363 15550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "54a1953220c04b03a5dbde97fc72135e"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-dssx"
server_key: "93adb0bd950247538c614a5528f0558d"
server_key_iv: "4a607f954eea74530b9845baa9b41c6c"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.062007 15550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:30:08.065887 15564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.067413 15550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:30:08.067581 15550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "54a1953220c04b03a5dbde97fc72135e"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-dssx"
server_key: "93adb0bd950247538c614a5528f0558d"
server_key_iv: "4a607f954eea74530b9845baa9b41c6c"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.067785 15550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:08.120501 15550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:08.121450 15550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:08.121742 15550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:08.122717 15550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:08.124204 15550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:08.124306 15550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.124395 15550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:08.124452 15550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.158516 15550 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:36655
I20260929 16:30:08.158561 15676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:36655 every 8 connection(s)
I20260929 16:30:08.159873 15550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:08.163194 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 15550
I20260929 16:30:08.163405 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:08.163836 6050 external_mini_cluster.cc:1428] Setting key b9879a97bf286d79a64b607f02da7fa7
I20260929 16:30:08.166594 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:33825
--builtin_ntp_servers=127.5.232.148:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:30:08.173805 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.162810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:40967 (local address 127.5.232.190:33825)
0929 16:30:08.163019 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.163025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.164819 (+ 1794us) server_negotiation.cc:407] Connection header received
0929 16:30:08.166737 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.166743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.166848 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.166962 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.168133 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.168866 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.170617 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.171014 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.171164 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.172972 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:08.172979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:08.172984 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:08.173020 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:08.173055 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:08.173069 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.173148 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.173488 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.173576 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:30:08.175488 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.163470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:33825 (local address 127.5.232.130:40967)
0929 16:30:08.164628 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.164677 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:08.166434 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:08.167229 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:08.167239 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:08.167349 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:08.167954 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.167971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.169028 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.169035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.170445 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.170460 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.171265 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.171272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.171533 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.172506 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:08.172512 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:08.172575 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:08.172697 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:08.173183 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:08.173195 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:08.173204 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.173485 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.173984 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":348,"threads_started":1}
I20260929 16:30:08.177104 15677 heartbeater.cc:347] Connected to a master server at 127.5.232.190:33825
I20260929 16:30:08.177500 15677 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:08.178335 15677 heartbeater.cc:511] Master 127.5.232.190:33825 requested a full tablet report, sending...
I20260929 16:30:08.179900 15360 ts_manager.cc:194] Registered new tserver with Master: 54a1953220c04b03a5dbde97fc72135e (127.5.232.130:36655)
I20260929 16:30:08.180734 15360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.130:40967
W20260929 16:30:08.513585 15680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:08.513936 15680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:08.514096 15680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:08.521539 15680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:08.521685 15680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:08.521858 15680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:30:08.531207 15680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:33825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:08.532598 15680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:08.534389 15680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:08.547961 15685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:08.547891 15686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:08.547937 15688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:08.548699 15680 server_base.cc:1034] running on GCE node
I20260929 16:30:08.549307 15680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:08.550482 15680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:08.551723 15680 hybrid_clock.cc:648] HybridClock initialized: now 1790699408551680 us; error 62 us; skew 500 ppm
I20260929 16:30:08.552065 15680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:30:08.554517 15680 webserver.cc:460] Webserver started at http://127.5.232.131:45907/ using document root <none> and password file <none>
I20260929 16:30:08.555277 15680 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:08.555369 15680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:08.555656 15680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:08.558772 15680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "6b4ec91320f941f4b84862e19425ebb0"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-dssx"
server_key: "b718b6275e396aa8fedeafbb8c2e0fc6"
server_key_iv: "a58ac7127efece3e22180e0c343f5686"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.559584 15680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "6b4ec91320f941f4b84862e19425ebb0"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-dssx"
server_key: "b718b6275e396aa8fedeafbb8c2e0fc6"
server_key_iv: "a58ac7127efece3e22180e0c343f5686"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.565153 15680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:30:08.569145 15694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.570798 15680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:08.571035 15680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "6b4ec91320f941f4b84862e19425ebb0"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-dssx"
server_key: "b718b6275e396aa8fedeafbb8c2e0fc6"
server_key_iv: "a58ac7127efece3e22180e0c343f5686"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.571297 15680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:08.589139 15680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:08.590458 15680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:08.590834 15680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:08.591984 15680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:08.593693 15680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:08.593907 15680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.594082 15680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:08.594179 15680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.627571 15680 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:36989
I20260929 16:30:08.627619 15806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:36989 every 8 connection(s)
I20260929 16:30:08.628947 15680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:08.632807 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 15680
I20260929 16:30:08.633001 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:08.633543 6050 external_mini_cluster.cc:1428] Setting key 9d329c0d74134082d4f48591a60425ec
I20260929 16:30:08.642222 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.631672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:33629 (local address 127.5.232.190:33825)
0929 16:30:08.631859 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.631863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.633572 (+ 1709us) server_negotiation.cc:407] Connection header received
0929 16:30:08.635688 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.635693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.635780 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.635901 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.637087 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.637876 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.639440 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.639847 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.639991 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.641312 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:08.641317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:08.641320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:08.641340 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:08.641368 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:08.641429 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.641513 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.641755 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.641858 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:30:08.643309 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.632230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:33825 (local address 127.5.232.131:33629)
0929 16:30:08.633337 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.633423 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:08.635369 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:08.636196 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:08.636209 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:08.636332 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:08.636925 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.636943 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.638033 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.638038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.639293 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.639305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.640072 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.640079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.640320 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.640983 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:08.640985 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:08.641026 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:08.641106 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:08.641513 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:08.641527 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:08.641538 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.641749 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.642180 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":317,"threads_started":1}
I20260929 16:30:08.644773 15807 heartbeater.cc:347] Connected to a master server at 127.5.232.190:33825
I20260929 16:30:08.645082 15807 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:08.645951 15807 heartbeater.cc:511] Master 127.5.232.190:33825 requested a full tablet report, sending...
I20260929 16:30:08.647666 15360 ts_manager.cc:194] Registered new tserver with Master: 6b4ec91320f941f4b84862e19425ebb0 (127.5.232.131:36989)
I20260929 16:30:08.648620 15360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.131:33629
I20260929 16:30:08.649204 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:30:08.656515 15548 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:08.654653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.5.232.190:33825)
0929 16:30:08.654865 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.654873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.655343 (+ 470us) server_negotiation.cc:407] Connection header received
0929 16:30:08.655562 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.655567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.655652 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.655765 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.656243 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46044: BlockingRecv error: recv got EOF from 127.0.0.1:46044 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:30:08.657135 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15419
I20260929 16:30:08.666551 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.658178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.5.232.129:44567)
0929 16:30:08.658769 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.658778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.658809 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:30:08.658875 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.658883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.659164 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.659352 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.659701 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.661325 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.662747 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.663654 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.663796 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.664942 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:08.664954 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:08.664970 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:08.665721 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:08.665770 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:08.665791 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.665892 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.666140 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.666259 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":307,"threads_started":1}
I20260929 16:30:08.667860 15541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:08.749579 15546 heartbeater.cc:503] Master 127.5.232.190:33825 was elected leader, sending a full tablet report...
I20260929 16:30:08.750900 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 15419
W20260929 16:30:08.767392 15347 connection.cc:535] server connection from 127.5.232.129:40867 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:30:08.767513 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15550
I20260929 16:30:08.775964 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.768429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.5.232.130:36655)
0929 16:30:08.769026 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.769040 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.769069 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:30:08.769132 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.769138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.769364 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.769600 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.769950 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.770977 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.772505 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.773102 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.773217 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.774409 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:08.774418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:08.774433 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:08.775071 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:08.775127 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:08.775184 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.775302 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.775421 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.775573 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":309,"threads_started":1}
I20260929 16:30:08.777241 15672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:08.860107 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 15550
I20260929 16:30:08.876232 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15680
I20260929 16:30:08.884722 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.877238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.5.232.131:36989)
0929 16:30:08.877948 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.877955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.877989 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:30:08.878052 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.878059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.878287 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.878435 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.878833 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.879875 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.881135 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.881827 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.881935 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.883077 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:08.883086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:08.883101 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:08.883751 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:08.883801 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:08.883821 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.883935 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.884182 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.884358 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":403,"threads_started":1}
I20260929 16:30:08.886147 15802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:08.970566 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 15680
I20260929 16:30:08.987097 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15329
I20260929 16:30:08.988441 15390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:09.091034 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 15329
2026-09-29T16:30:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2524 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T16:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:09Z Disabled control of system clock
I20260929 16:30:09.163690 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:37601
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:37601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:30:09.508116 15826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:09.508414 15826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:09.508507 15826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:09.515762 15826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:09.515872 15826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:09.515918 15826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:09.515969 15826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:09.516016 15826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:09.525121 15826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:37601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:37601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:09.526520 15826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:09.528312 15826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:09.540056 15832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:09.540083 15831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:09.540179 15834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:09.541024 15826 server_base.cc:1034] running on GCE node
I20260929 16:30:09.542096 15826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:09.543808 15826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:09.545017 15826 hybrid_clock.cc:648] HybridClock initialized: now 1790699409544983 us; error 35 us; skew 500 ppm
I20260929 16:30:09.545444 15826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:30:09.547816 15826 webserver.cc:460] Webserver started at http://127.5.232.190:38533/ using document root <none> and password file <none>
I20260929 16:30:09.548542 15826 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:09.548658 15826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:09.548990 15826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:09.553236 15826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "b24223f2a63b404698528e1261f15cd6"
format_stamp: "Formatted at 2026-09-29 16:30:09 on dist-test-slave-dssx"
server_key: "f2acca2859db67bb9cdc9d59629fcd1c"
server_key_iv: "0b27dfe1a6be810697630e9f42652349"
server_key_version: "encryptionkey@0"
I20260929 16:30:09.554131 15826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "b24223f2a63b404698528e1261f15cd6"
format_stamp: "Formatted at 2026-09-29 16:30:09 on dist-test-slave-dssx"
server_key: "f2acca2859db67bb9cdc9d59629fcd1c"
server_key_iv: "0b27dfe1a6be810697630e9f42652349"
server_key_version: "encryptionkey@0"
I20260929 16:30:09.559388 15826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 16:30:09.563006 15840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:09.564391 15826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:09.564605 15826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "b24223f2a63b404698528e1261f15cd6"
format_stamp: "Formatted at 2026-09-29 16:30:09 on dist-test-slave-dssx"
server_key: "f2acca2859db67bb9cdc9d59629fcd1c"
server_key_iv: "0b27dfe1a6be810697630e9f42652349"
server_key_version: "encryptionkey@0"
I20260929 16:30:09.564826 15826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:09.570653 15826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:09.571444 15826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:09.571743 15826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:09.589893 15826 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:37601
I20260929 16:30:09.589910 15891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:37601 every 8 connection(s)
I20260929 16:30:09.591310 15826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:09.595113 15892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:09.596993 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 15826
I20260929 16:30:09.597173 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:30:09.598017 6050 external_mini_cluster.cc:1428] Setting key d886e00273f14d91b6f6b77348b5e736
I20260929 16:30:09.606706 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.600169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.5.232.190:37601)
0929 16:30:09.601357 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.601371 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.601466 (+ 95us) server_negotiation.cc:407] Connection header received
0929 16:30:09.603058 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.603081 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.603510 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.603854 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:09.604999 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:09.605005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:09.605018 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:09.605035 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:09.605106 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:09.605123 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:09.605294 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.605529 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.605813 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":405,"threads_started":1}
I20260929 16:30:09.608016 15892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6: Bootstrap starting.
I20260929 16:30:09.611231 15892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:09.612231 15892 log.cc:824] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:09.614890 15892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6: No bootstrap required, opened a new log
I20260929 16:30:09.622303 15892 raft_consensus.cc:348] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24223f2a63b404698528e1261f15cd6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37601 } }
I20260929 16:30:09.622704 15892 raft_consensus.cc:374] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:09.622808 15892 raft_consensus.cc:729] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24223f2a63b404698528e1261f15cd6, State: Initialized, Role: FOLLOWER
I20260929 16:30:09.623420 15892 consensus_queue.cc:260] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24223f2a63b404698528e1261f15cd6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37601 } }
I20260929 16:30:09.623667 15892 raft_consensus.cc:388] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:09.623805 15892 raft_consensus.cc:482] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:09.623979 15892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:09.626188 15892 raft_consensus.cc:504] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24223f2a63b404698528e1261f15cd6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37601 } }
I20260929 16:30:09.626744 15892 leader_election.cc:302] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b24223f2a63b404698528e1261f15cd6; no voters:
I20260929 16:30:09.627460 15892 leader_election.cc:288] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:09.627676 15897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:09.628448 15897 raft_consensus.cc:686] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [term 1 LEADER]: Becoming Leader. State: Replica: b24223f2a63b404698528e1261f15cd6, State: Running, Role: LEADER
I20260929 16:30:09.629222 15897 consensus_queue.cc:237] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24223f2a63b404698528e1261f15cd6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37601 } }
I20260929 16:30:09.629765 15892 sys_catalog.cc:564] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:09.632207 15898 sys_catalog.cc:455] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b24223f2a63b404698528e1261f15cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24223f2a63b404698528e1261f15cd6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37601 } } }
I20260929 16:30:09.632391 15899 sys_catalog.cc:455] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b24223f2a63b404698528e1261f15cd6. Latest consensus state: current_term: 1 leader_uuid: "b24223f2a63b404698528e1261f15cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24223f2a63b404698528e1261f15cd6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37601 } } }
I20260929 16:30:09.632630 15898 sys_catalog.cc:458] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:09.632688 15899 sys_catalog.cc:458] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:09.635478 15906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:09.639953 15906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:09.648537 15906 catalog_manager.cc:1347] Generated new cluster ID: b16ef56ad61540a8b0c3e783d72ea657
I20260929 16:30:09.648662 15906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:09.668177 15906 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:09.668691 15906 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:09.678043 15906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b24223f2a63b404698528e1261f15cd6: Generated new TSK 0
I20260929 16:30:09.678960 15906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:09.690039 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37601
--builtin_ntp_servers=127.5.232.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:30:10.053648 15916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:10.053934 15916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:10.054121 15916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:10.061318 15916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:10.061450 15916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:10.061632 15916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:30:10.071197 15916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:10.072484 15916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:10.074277 15916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:10.087942 15921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:10.087989 15922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:10.088166 15924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:10.088722 15916 server_base.cc:1034] running on GCE node
I20260929 16:30:10.089352 15916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:10.090457 15916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:10.091730 15916 hybrid_clock.cc:648] HybridClock initialized: now 1790699410091679 us; error 42 us; skew 500 ppm
I20260929 16:30:10.092116 15916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:30:10.094722 15916 webserver.cc:460] Webserver started at http://127.5.232.129:40671/ using document root <none> and password file <none>
I20260929 16:30:10.095494 15916 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:10.095645 15916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:10.096007 15916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:10.099093 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "9462ed7edce44fae98b8117c93da05e3"
format_stamp: "Formatted at 2026-09-29 16:30:10 on dist-test-slave-dssx"
server_key: "7d3baf9a8b3a961ee176d66ed10132ba"
server_key_iv: "eaef7d88f82c3f5f019d3e83dc119eee"
server_key_version: "encryptionkey@0"
I20260929 16:30:10.099973 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "9462ed7edce44fae98b8117c93da05e3"
format_stamp: "Formatted at 2026-09-29 16:30:10 on dist-test-slave-dssx"
server_key: "7d3baf9a8b3a961ee176d66ed10132ba"
server_key_iv: "eaef7d88f82c3f5f019d3e83dc119eee"
server_key_version: "encryptionkey@0"
I20260929 16:30:10.105876 15916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:30:10.110028 15930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:10.111557 15916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:30:10.111758 15916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "9462ed7edce44fae98b8117c93da05e3"
format_stamp: "Formatted at 2026-09-29 16:30:10 on dist-test-slave-dssx"
server_key: "7d3baf9a8b3a961ee176d66ed10132ba"
server_key_iv: "eaef7d88f82c3f5f019d3e83dc119eee"
server_key_version: "encryptionkey@0"
I20260929 16:30:10.112012 15916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:10.118021 15916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:10.118873 15916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:10.119164 15916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:10.120100 15916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:10.121634 15916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:10.121760 15916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:10.121891 15916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:10.121980 15916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:10.155102 15916 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:41479
I20260929 16:30:10.155157 16042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:41479 every 8 connection(s)
I20260929 16:30:10.156459 15916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:10.164860 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 15916
I20260929 16:30:10.165138 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:10.165470 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:10.159112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:46665 (local address 127.5.232.190:37601)
0929 16:30:10.159581 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:10.159585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:10.160663 (+ 1078us) server_negotiation.cc:407] Connection header received
0929 16:30:10.162681 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:10.162687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:10.162760 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:10.162894 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:10.164598 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:10.164604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:10.164607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:10.164625 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:10.164668 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:10.164679 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:10.164826 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:10.165054 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 16:30:10.165126 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":261,"threads_started":1}
I20260929 16:30:10.165956 6050 external_mini_cluster.cc:1428] Setting key 571185b0a110bc34cb5cfc44fb2b1890
I20260929 16:30:10.166648 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:10.159596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:37601 (local address 127.5.232.129:46665)
0929 16:30:10.160485 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:10.160525 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:10.162353 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:10.163160 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:10.163170 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:10.163452 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:10.164192 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:10.164194 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:10.164235 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:10.164339 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:10.164794 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:10.164805 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:10.164813 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:30:10.165167 (+ 354us) client_negotiation.cc:240] Negotiation successful
0929 16:30:10.165509 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":251,"threads_started":1}
I20260929 16:30:10.169147 16043 heartbeater.cc:347] Connected to a master server at 127.5.232.190:37601
I20260929 16:30:10.169545 16043 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:10.170569 16043 heartbeater.cc:511] Master 127.5.232.190:37601 requested a full tablet report, sending...
I20260929 16:30:10.170706 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37601
--builtin_ntp_servers=127.5.232.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:30:10.173211 15857 ts_manager.cc:194] Registered new tserver with Master: 9462ed7edce44fae98b8117c93da05e3 (127.5.232.129:41479)
W20260929 16:30:10.512831 16047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:10.513195 16047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:10.513365 16047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:10.520985 16047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:10.521150 16047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:10.521348 16047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:30:10.531523 16047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:10.532908 16047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:10.535068 16047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:10.549021 16052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:10.549129 16053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:10.549552 16055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:10.550318 16047 server_base.cc:1034] running on GCE node
I20260929 16:30:10.550930 16047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:10.551916 16047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:10.553184 16047 hybrid_clock.cc:648] HybridClock initialized: now 1790699410553135 us; error 37 us; skew 500 ppm
I20260929 16:30:10.553648 16047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:30:10.556291 16047 webserver.cc:460] Webserver started at http://127.5.232.130:38811/ using document root <none> and password file <none>
I20260929 16:30:10.557077 16047 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:10.557173 16047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:10.557601 16047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:10.560640 16047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "0ed2b8db7a2f4ae9aa4ec91cf1dc4d75"
format_stamp: "Formatted at 2026-09-29 16:30:10 on dist-test-slave-dssx"
server_key: "74cc65837963f938d8b1890d659d7ca5"
server_key_iv: "069bef740b15125ecdda5fd19c249ab5"
server_key_version: "encryptionkey@0"
I20260929 16:30:10.561672 16047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "0ed2b8db7a2f4ae9aa4ec91cf1dc4d75"
format_stamp: "Formatted at 2026-09-29 16:30:10 on dist-test-slave-dssx"
server_key: "74cc65837963f938d8b1890d659d7ca5"
server_key_iv: "069bef740b15125ecdda5fd19c249ab5"
server_key_version: "encryptionkey@0"
I20260929 16:30:10.567562 16047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:30:10.571565 16061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:10.573060 16047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:30:10.573268 16047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "0ed2b8db7a2f4ae9aa4ec91cf1dc4d75"
format_stamp: "Formatted at 2026-09-29 16:30:10 on dist-test-slave-dssx"
server_key: "74cc65837963f938d8b1890d659d7ca5"
server_key_iv: "069bef740b15125ecdda5fd19c249ab5"
server_key_version: "encryptionkey@0"
I20260929 16:30:10.573575 16047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:10.579704 16047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:10.580565 16047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:10.580857 16047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:10.581884 16047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:10.583367 16047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:10.583516 16047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:10.583647 16047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:10.583732 16047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:10.619954 16047 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:32781
I20260929 16:30:10.620000 16173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:32781 every 8 connection(s)
I20260929 16:30:10.621335 16047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:10.626581 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 16047
I20260929 16:30:10.626751 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:10.627291 6050 external_mini_cluster.cc:1428] Setting key 5ee64fa95349d312f29ba3274fb7568f
I20260929 16:30:10.630369 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37601
--builtin_ntp_servers=127.5.232.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:30:10.630481 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:10.624077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:44447 (local address 127.5.232.190:37601)
0929 16:30:10.624250 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:10.624255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:10.625692 (+ 1437us) server_negotiation.cc:407] Connection header received
0929 16:30:10.628044 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:10.628050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:10.628128 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:10.628245 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:10.629773 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:10.629777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:10.629781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:10.629796 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:10.629826 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:10.629837 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:10.629965 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:10.630175 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 16:30:10.630223 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:30:10.631845 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:10.624502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:37601 (local address 127.5.232.130:44447)
0929 16:30:10.625531 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:10.625578 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:10.627714 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:10.628485 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:10.628493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:10.628594 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:10.629359 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:10.629362 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:10.629432 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:10.629548 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:10.629949 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:10.629960 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:10.629969 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:30:10.630289 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 16:30:10.630574 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":277,"threads_started":1}
I20260929 16:30:10.633294 16174 heartbeater.cc:347] Connected to a master server at 127.5.232.190:37601
I20260929 16:30:10.633572 16174 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:10.634265 16174 heartbeater.cc:511] Master 127.5.232.190:37601 requested a full tablet report, sending...
I20260929 16:30:10.636024 15857 ts_manager.cc:194] Registered new tserver with Master: 0ed2b8db7a2f4ae9aa4ec91cf1dc4d75 (127.5.232.130:32781)
W20260929 16:30:10.973464 16177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:10.973785 16177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:10.973938 16177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:10.981685 16177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:10.981840 16177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:10.982035 16177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:30:10.991592 16177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:10.993002 16177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:10.994925 16177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:11.008256 16182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:11.008551 16183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:11.008590 16185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:11.009879 16177 server_base.cc:1034] running on GCE node
I20260929 16:30:11.010507 16177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:11.011505 16177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:11.012738 16177 hybrid_clock.cc:648] HybridClock initialized: now 1790699411012677 us; error 54 us; skew 500 ppm
I20260929 16:30:11.013075 16177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:30:11.015515 16177 webserver.cc:460] Webserver started at http://127.5.232.131:44593/ using document root <none> and password file <none>
I20260929 16:30:11.016328 16177 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:11.016466 16177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:11.016798 16177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:11.019851 16177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "a28ec0f3d8914c7c8f0c7675b0fe3451"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-dssx"
server_key: "19a5080b4aa92220067aa97d1d016d9a"
server_key_iv: "b254d09f1ec306a3d9bf1067aefd65d7"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.020718 16177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "a28ec0f3d8914c7c8f0c7675b0fe3451"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-dssx"
server_key: "19a5080b4aa92220067aa97d1d016d9a"
server_key_iv: "b254d09f1ec306a3d9bf1067aefd65d7"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.026456 16177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:30:11.030340 16191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.031984 16177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:30:11.032178 16177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "a28ec0f3d8914c7c8f0c7675b0fe3451"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-dssx"
server_key: "19a5080b4aa92220067aa97d1d016d9a"
server_key_iv: "b254d09f1ec306a3d9bf1067aefd65d7"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.032415 16177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:11.038648 16177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:11.039507 16177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:11.039789 16177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:11.040729 16177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:11.042268 16177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:11.042392 16177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.042533 16177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:11.042620 16177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.076262 16177 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:42471
I20260929 16:30:11.076357 16303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:42471 every 8 connection(s)
I20260929 16:30:11.077817 16177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:11.086090 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.080584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:53389 (local address 127.5.232.190:37601)
0929 16:30:11.080756 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:11.080761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:11.082263 (+ 1502us) server_negotiation.cc:407] Connection header received
0929 16:30:11.083967 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:11.083972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:11.084026 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:11.084117 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:11.085452 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:11.085467 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:11.085470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:11.085487 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:11.085515 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:11.085528 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:11.085626 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:11.085806 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 16:30:11.085857 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:30:11.086374 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 16177
I20260929 16:30:11.086548 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:11.087107 6050 external_mini_cluster.cc:1428] Setting key 338f22216083080a2c508357372b47b0
I20260929 16:30:11.087450 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.081148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:37601 (local address 127.5.232.131:53389)
0929 16:30:11.082118 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:11.082162 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:11.083692 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:11.084337 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:11.084346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:11.084446 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:11.085087 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:11.085089 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:11.085129 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:11.085218 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:11.085617 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:11.085625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:11.085631 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:30:11.085858 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 16:30:11.086153 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":327,"threads_started":1}
I20260929 16:30:11.088973 16304 heartbeater.cc:347] Connected to a master server at 127.5.232.190:37601
I20260929 16:30:11.089313 16304 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:11.090067 16304 heartbeater.cc:511] Master 127.5.232.190:37601 requested a full tablet report, sending...
I20260929 16:30:11.091423 15856 ts_manager.cc:194] Registered new tserver with Master: a28ec0f3d8914c7c8f0c7675b0fe3451 (127.5.232.131:42471)
I20260929 16:30:11.103157 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:30:11.109726 16045 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:11.108005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.5.232.190:37601)
0929 16:30:11.108201 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:11.108206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:11.108775 (+ 569us) server_negotiation.cc:407] Connection header received
0929 16:30:11.108924 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:11.108928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:11.108979 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:11.109071 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:11.109345 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44352: BlockingRecv error: recv got EOF from 127.0.0.1:44352 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:30:11.110363 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15916
I20260929 16:30:11.113898 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.111301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.5.232.129:41479)
0929 16:30:11.111804 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:11.111811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:11.111837 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:30:11.111897 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:11.111903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:11.112134 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:11.112264 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:11.113145 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:11.113152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:11.113164 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:11.113181 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:11.113224 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:11.113239 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:11.113341 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:11.113555 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 16:30:11.113637 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":245,"threads_started":1}
I20260929 16:30:11.115178 16038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:11.196171 16043 heartbeater.cc:503] Master 127.5.232.190:37601 was elected leader, sending a full tablet report...
I20260929 16:30:11.197793 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 15916
W20260929 16:30:11.211787 15843 connection.cc:535] server connection from 127.5.232.129:46665 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:30:11.212070 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 16047
I20260929 16:30:11.215456 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.213003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.5.232.130:32781)
0929 16:30:11.213552 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:11.213560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:11.213587 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:30:11.213650 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:11.213656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:11.213877 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:11.214005 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:11.214889 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:11.214895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:11.214908 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:11.214919 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:11.214963 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:11.214976 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:11.215079 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:11.215154 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 16:30:11.215237 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":202,"threads_started":1}
I20260929 16:30:11.216681 16169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:11.297034 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 16047
I20260929 16:30:11.314914 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 16177
I20260929 16:30:11.319011 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.315941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.5.232.131:42471)
0929 16:30:11.316545 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:11.316552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:11.316580 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:30:11.316643 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:11.316650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:11.316867 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:11.317044 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:11.318270 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:11.318276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:11.318289 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:11.318304 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:11.318343 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:11.318358 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:11.318547 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:11.318622 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 16:30:11.318720 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":308,"threads_started":1}
I20260929 16:30:11.320254 16299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:11.402063 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 16177
I20260929 16:30:11.418139 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 15826
I20260929 16:30:11.419310 15887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:11.522486 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 15826
2026-09-29T16:30:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2430 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:11 dist-test-slave-dssx krb5kdc[16322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:11 dist-test-slave-dssx krb5kdc[16322](info): set up 2 sockets
Sep 29 16:30:11 dist-test-slave-dssx krb5kdc[16322](info): commencing operation
krb5kdc: starting...
W20260929 16:30:12.638310 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:30:12 dist-test-slave-dssx krb5kdc[16322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:30:12.868752 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:37905
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:37905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:13.195523 16335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:13.195839 16335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:13.195931 16335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:13.203167 16335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:13.203289 16335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:13.203334 16335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:13.203383 16335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:13.203428 16335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:13.213435 16335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:37905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:13.214747 16335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:13.216141 16335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:13.227710 16340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:13.227737 16343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:13.227774 16341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:13.228595 16335 server_base.cc:1034] running on GCE node
I20260929 16:30:13.229426 16335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:13.230895 16335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:13.232091 16335 hybrid_clock.cc:648] HybridClock initialized: now 1790699413232053 us; error 34 us; skew 500 ppm
I20260929 16:30:13.232434 16335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:30:13 dist-test-slave-dssx krb5kdc[16322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699413, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:13.235670 16335 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:30:13.237182 16335 webserver.cc:460] Webserver started at http://127.5.232.190:35517/ using document root <none> and password file <none>
I20260929 16:30:13.237941 16335 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:13.238089 16335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:13.238389 16335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:13.241357 16335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "48b3c91c08a74e47852d061ba4337b01"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-dssx"
server_key: "55fc5183f0bd57ed272cc75432dbb083"
server_key_iv: "dc17eba67bdcacef9460fbc8729a242c"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.242201 16335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "48b3c91c08a74e47852d061ba4337b01"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-dssx"
server_key: "55fc5183f0bd57ed272cc75432dbb083"
server_key_iv: "dc17eba67bdcacef9460fbc8729a242c"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.247485 16335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:30:13.251047 16350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.252578 16335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:13.252801 16335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "48b3c91c08a74e47852d061ba4337b01"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-dssx"
server_key: "55fc5183f0bd57ed272cc75432dbb083"
server_key_iv: "dc17eba67bdcacef9460fbc8729a242c"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.253021 16335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:13.274987 16335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:13.279767 16335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:13.280092 16335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:13.298810 16335 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:37905
I20260929 16:30:13.298832 16401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:37905 every 8 connection(s)
I20260929 16:30:13.300243 16335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:13.301661 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 16335
I20260929 16:30:13.301872 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:30:13.302526 6050 external_mini_cluster.cc:1428] Setting key 7fd67ba9da977dc70d06ed7e18f19aa9
I20260929 16:30:13.304615 16402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:30:13 dist-test-slave-dssx krb5kdc[16322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:30:13.318085 16402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01: Bootstrap starting.
I20260929 16:30:13.322438 16402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:13.323721 16402 log.cc:824] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:13.326661 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.304613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.5.232.190:37905)
0929 16:30:13.305489 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.305505 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.305556 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:30:13.307680 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.307709 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.308244 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.308726 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:13.310117 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.312162 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.313446 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.314480 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.314670 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.318673 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:13.318683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:13.318706 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:13.318755 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:13.322639 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:13.323300 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:13.323308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:13.323316 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:13.323556 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:13.323952 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:13.323956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:13.323958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:13.324521 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:13.324861 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.325286 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.325730 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":318,"threads_started":1}
I20260929 16:30:13.326752 16402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01: No bootstrap required, opened a new log
I20260929 16:30:13.334199 16402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3c91c08a74e47852d061ba4337b01" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37905 } }
I20260929 16:30:13.334609 16402 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:13.334718 16402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b3c91c08a74e47852d061ba4337b01, State: Initialized, Role: FOLLOWER
I20260929 16:30:13.335387 16402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3c91c08a74e47852d061ba4337b01" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37905 } }
I20260929 16:30:13.335664 16402 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:13.335806 16402 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:13.336001 16402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:13.338179 16402 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3c91c08a74e47852d061ba4337b01" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37905 } }
I20260929 16:30:13.338704 16402 leader_election.cc:302] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b3c91c08a74e47852d061ba4337b01; no voters:
I20260929 16:30:13.339313 16402 leader_election.cc:288] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:13.339468 16407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:13.340191 16407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [term 1 LEADER]: Becoming Leader. State: Replica: 48b3c91c08a74e47852d061ba4337b01, State: Running, Role: LEADER
I20260929 16:30:13.340884 16407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3c91c08a74e47852d061ba4337b01" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37905 } }
I20260929 16:30:13.341601 16402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:13.343775 16408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48b3c91c08a74e47852d061ba4337b01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3c91c08a74e47852d061ba4337b01" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37905 } } }
I20260929 16:30:13.343750 16409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b3c91c08a74e47852d061ba4337b01. Latest consensus state: current_term: 1 leader_uuid: "48b3c91c08a74e47852d061ba4337b01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3c91c08a74e47852d061ba4337b01" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37905 } } }
I20260929 16:30:13.344156 16408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:13.344231 16409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:13.347128 16412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:13.351647 16412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:13.360378 16412 catalog_manager.cc:1347] Generated new cluster ID: f86462f1af0047c8a524aad7a6c9299f
I20260929 16:30:13.360505 16412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:13.382185 16412 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:13.384143 16412 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:13.393953 16412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01: Generated new TSK 0
I20260929 16:30:13.395015 16412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:30:13.458223 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37905
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:13.807514 16428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:13.807806 16428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:13.807973 16428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:13.815325 16428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:13.815436 16428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:13.815617 16428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:30:13.825289 16428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:13.826633 16428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:13.827951 16428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:13.840905 16433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:13.840960 16436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:13.840871 16434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:13.842072 16428 server_base.cc:1034] running on GCE node
I20260929 16:30:13.842792 16428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:13.843801 16428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:13.845032 16428 hybrid_clock.cc:648] HybridClock initialized: now 1790699413844975 us; error 48 us; skew 500 ppm
I20260929 16:30:13.845494 16428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:30:13 dist-test-slave-dssx krb5kdc[16322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699413, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:13.848709 16428 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:30:13.850318 16428 webserver.cc:460] Webserver started at http://127.5.232.129:39661/ using document root <none> and password file <none>
I20260929 16:30:13.851012 16428 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:13.851158 16428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:13.851452 16428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:13.854437 16428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "20094fd146aa4c208d37f9cf459ef260"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-dssx"
server_key: "dbbce74b3cafec299973c1cdc155b843"
server_key_iv: "29b4d700e2eff4877c20ba10d3e2c5b9"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.855219 16428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "20094fd146aa4c208d37f9cf459ef260"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-dssx"
server_key: "dbbce74b3cafec299973c1cdc155b843"
server_key_iv: "29b4d700e2eff4877c20ba10d3e2c5b9"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.860428 16428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:30:13.863938 16443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.865278 16428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:30:13.865507 16428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "20094fd146aa4c208d37f9cf459ef260"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-dssx"
server_key: "dbbce74b3cafec299973c1cdc155b843"
server_key_iv: "29b4d700e2eff4877c20ba10d3e2c5b9"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.865713 16428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:13.904282 16428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:13.908910 16428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:13.909229 16428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:13.910290 16428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:13.911680 16428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:13.911785 16428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.911931 16428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:13.912002 16428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.946349 16428 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:39141
I20260929 16:30:13.946444 16555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:39141 every 8 connection(s)
I20260929 16:30:13.947774 16428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:13.954380 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 16428
I20260929 16:30:13.954596 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:13.955216 6050 external_mini_cluster.cc:1428] Setting key f196cd611685c603b359ebe7eb7f9269
Sep 29 16:30:13 dist-test-slave-dssx krb5kdc[16322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699413, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:30:13.969810 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.950468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:52421 (local address 127.5.232.190:37905)
0929 16:30:13.950924 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.950929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.951986 (+ 1057us) server_negotiation.cc:407] Connection header received
0929 16:30:13.953870 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.953876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.953968 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.954104 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:13.955768 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.956557 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.958317 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.958721 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.959010 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.962890 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:13.962896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:13.962899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:13.962936 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:13.966000 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:13.967088 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:13.967093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:13.967095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:13.967228 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:13.967763 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:13.967766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:13.967768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:13.968032 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:13.968146 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.969235 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.969499 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":237,"threads_started":1}
I20260929 16:30:13.971292 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.951057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:37905 (local address 127.5.232.129:52421)
0929 16:30:13.951801 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.951840 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.953594 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.954345 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.954354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.954970 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:13.955601 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.955619 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.956733 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.956737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.958165 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.958177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.959108 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.959118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.959365 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.960093 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:13.960116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:13.962619 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:13.966180 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:13.966193 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:13.966218 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:13.966883 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:13.967359 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:13.967366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:13.967373 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:13.967641 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:13.968176 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:13.968188 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:13.968707 (+ 519us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.969069 (+ 362us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.969715 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":295,"threads_started":1}
I20260929 16:30:13.972997 16556 heartbeater.cc:347] Connected to a master server at 127.5.232.190:37905
I20260929 16:30:13.973354 16556 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:13.974215 16556 heartbeater.cc:511] Master 127.5.232.190:37905 requested a full tablet report, sending...
I20260929 16:30:13.976468 16367 ts_manager.cc:194] Registered new tserver with Master: 20094fd146aa4c208d37f9cf459ef260 (127.5.232.129:39141)
I20260929 16:30:13.978082 16367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:52421
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:30:14.009824 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37905
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:14.341027 16562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:14.341383 16562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:14.341599 16562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:14.349216 16562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:14.349383 16562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:14.349620 16562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:30:14.359664 16562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:14.361027 16562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:14.362450 16562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:14.376094 16568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:14.376127 16570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:14.376794 16562 server_base.cc:1034] running on GCE node
W20260929 16:30:14.376893 16567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:14.377641 16562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:14.378602 16562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:14.379832 16562 hybrid_clock.cc:648] HybridClock initialized: now 1790699414379791 us; error 37 us; skew 500 ppm
I20260929 16:30:14.380163 16562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:30:14 dist-test-slave-dssx krb5kdc[16322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:14.383373 16562 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:30:14.384785 16562 webserver.cc:460] Webserver started at http://127.5.232.130:45203/ using document root <none> and password file <none>
I20260929 16:30:14.385560 16562 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:14.385710 16562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:14.386018 16562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:14.388926 16562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "56d2cea82c834ac4b20fb4c10bf20084"
format_stamp: "Formatted at 2026-09-29 16:30:14 on dist-test-slave-dssx"
server_key: "925603c5b622c65f558957f7b67960f7"
server_key_iv: "36a0dec6a84dd0ae78abc5306d111d03"
server_key_version: "encryptionkey@0"
I20260929 16:30:14.389801 16562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "56d2cea82c834ac4b20fb4c10bf20084"
format_stamp: "Formatted at 2026-09-29 16:30:14 on dist-test-slave-dssx"
server_key: "925603c5b622c65f558957f7b67960f7"
server_key_iv: "36a0dec6a84dd0ae78abc5306d111d03"
server_key_version: "encryptionkey@0"
I20260929 16:30:14.395257 16562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:30:14.399037 16577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:14.400563 16562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:14.400806 16562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "56d2cea82c834ac4b20fb4c10bf20084"
format_stamp: "Formatted at 2026-09-29 16:30:14 on dist-test-slave-dssx"
server_key: "925603c5b622c65f558957f7b67960f7"
server_key_iv: "36a0dec6a84dd0ae78abc5306d111d03"
server_key_version: "encryptionkey@0"
I20260929 16:30:14.401041 16562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:14.423764 16562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:14.427831 16562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:14.428128 16562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:14.429114 16562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:14.430541 16562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:14.430661 16562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:14.430816 16562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:14.430889 16562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:14.467458 16562 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:44575
I20260929 16:30:14.467525 16689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:44575 every 8 connection(s)
I20260929 16:30:14.469339 16562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:14.474807 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 16562
I20260929 16:30:14.474999 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:14.475458 6050 external_mini_cluster.cc:1428] Setting key b87c29ef9c08ec757fa37ddd9c534add
Sep 29 16:30:14 dist-test-slave-dssx krb5kdc[16322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:30:14.496785 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:14.472772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:37685 (local address 127.5.232.190:37905)
0929 16:30:14.473220 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:14.473225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:14.477211 (+ 3986us) server_negotiation.cc:407] Connection header received
0929 16:30:14.477279 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:14.477286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:14.477402 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:14.477517 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:14.479805 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:14.481002 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:14.483400 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:14.483992 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:14.484109 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:14.488835 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:14.488843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:14.488848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:14.488888 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:14.492551 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:14.493628 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:14.493634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:14.493637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:14.493830 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:14.494382 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:14.494388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:14.494393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:14.494781 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:14.494935 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:14.496024 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0929 16:30:14.496292 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":229,"threads_started":1}
I20260929 16:30:14.498093 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:14.473235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:37905 (local address 127.5.232.130:37685)
0929 16:30:14.474118 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:14.474168 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:14.476811 (+ 2643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:14.477832 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:14.477845 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:14.478685 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:14.479567 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:14.479614 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:14.481199 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:14.481203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:14.483210 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:14.483226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:14.484407 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:14.484414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:14.484709 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:14.485547 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:14.485577 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:14.488513 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:14.492743 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:14.492755 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:14.492778 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:14.493485 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:14.493981 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:14.493988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:14.493992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:14.494240 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:14.495012 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:14.495024 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:14.495516 (+ 492us) client_negotiation.cc:769] Sending connection context
0929 16:30:14.495873 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 16:30:14.496513 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":354,"threads_started":1}
I20260929 16:30:14.499980 16690 heartbeater.cc:347] Connected to a master server at 127.5.232.190:37905
I20260929 16:30:14.500363 16690 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:14.501408 16690 heartbeater.cc:511] Master 127.5.232.190:37905 requested a full tablet report, sending...
I20260929 16:30:14.503417 16367 ts_manager.cc:194] Registered new tserver with Master: 56d2cea82c834ac4b20fb4c10bf20084 (127.5.232.130:44575)
I20260929 16:30:14.504767 16367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:37685
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:30:14.536130 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37905
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:14.872259 16696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:14.872771 16696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:14.872983 16696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:14.880865 16696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:14.881001 16696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:14.881117 16696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:30:14.891642 16696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:30:14.892958 16696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:14.894399 16696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:14.908843 16701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:14.908794 16704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:14.908804 16702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:14.909536 16696 server_base.cc:1034] running on GCE node
I20260929 16:30:14.910310 16696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:14.911451 16696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:14.912732 16696 hybrid_clock.cc:648] HybridClock initialized: now 1790699414912678 us; error 45 us; skew 500 ppm
I20260929 16:30:14.913125 16696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:30:14 dist-test-slave-dssx krb5kdc[16322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:14.916831 16696 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:30:14.918671 16696 webserver.cc:460] Webserver started at http://127.5.232.131:43735/ using document root <none> and password file <none>
I20260929 16:30:14.919596 16696 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:14.919773 16696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:14.920110 16696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:14.923512 16696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "3b9f2892f08b442381f7af2fdf9f5d77"
format_stamp: "Formatted at 2026-09-29 16:30:14 on dist-test-slave-dssx"
server_key: "461cb927a1279b8f12efd90638b7a8b8"
server_key_iv: "0612cfc075fc29c19335dbc2216d4ebd"
server_key_version: "encryptionkey@0"
I20260929 16:30:14.924428 16696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "3b9f2892f08b442381f7af2fdf9f5d77"
format_stamp: "Formatted at 2026-09-29 16:30:14 on dist-test-slave-dssx"
server_key: "461cb927a1279b8f12efd90638b7a8b8"
server_key_iv: "0612cfc075fc29c19335dbc2216d4ebd"
server_key_version: "encryptionkey@0"
I20260929 16:30:14.930467 16696 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:30:14.934711 16711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:14.936285 16696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:30:14.936499 16696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "3b9f2892f08b442381f7af2fdf9f5d77"
format_stamp: "Formatted at 2026-09-29 16:30:14 on dist-test-slave-dssx"
server_key: "461cb927a1279b8f12efd90638b7a8b8"
server_key_iv: "0612cfc075fc29c19335dbc2216d4ebd"
server_key_version: "encryptionkey@0"
I20260929 16:30:14.936748 16696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:14.960505 16696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:14.965624 16696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:14.965972 16696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:14.967099 16696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:14.968777 16696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:14.968906 16696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:14.969084 16696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:14.969170 16696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:14.981364 16556 heartbeater.cc:503] Master 127.5.232.190:37905 was elected leader, sending a full tablet report...
I20260929 16:30:15.002689 16696 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:43749
I20260929 16:30:15.002736 16823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:43749 every 8 connection(s)
I20260929 16:30:15.004083 16696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:15.010288 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 16696
I20260929 16:30:15.010469 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:15.011021 6050 external_mini_cluster.cc:1428] Setting key 6c36930d8b0db1a538c5f32c129d8292
Sep 29 16:30:15 dist-test-slave-dssx krb5kdc[16322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:30:15.024111 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.006882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:43761 (local address 127.5.232.190:37905)
0929 16:30:15.007356 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.007363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.008381 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 16:30:15.010206 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.010212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.010304 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.010432 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:15.012443 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.013538 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.015288 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.015659 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.015756 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.019056 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:15.019060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:15.019063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:15.019090 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:15.021537 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.022184 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.022188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.022190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.022300 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.022663 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.022665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.022667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.022919 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:15.023028 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.023666 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.023828 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":223,"threads_started":1}
I20260929 16:30:15.025243 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.007519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:37905 (local address 127.5.232.131:43761)
0929 16:30:15.008217 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:15.008261 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:15.009924 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:15.010738 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:15.010750 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:15.011519 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:15.012265 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.012292 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.013673 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.013676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.015164 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.015175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.015844 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.015849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.016044 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.016727 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:15.016752 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:15.018876 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:15.021657 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:15.021663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:15.021678 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:15.022088 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:15.022388 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:15.022391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:15.022393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:15.022577 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:15.023022 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:15.023027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:15.023341 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 16:30:15.023595 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 16:30:15.023990 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":264,"threads_started":1}
I20260929 16:30:15.026867 16824 heartbeater.cc:347] Connected to a master server at 127.5.232.190:37905
I20260929 16:30:15.027155 16824 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:15.027858 16824 heartbeater.cc:511] Master 127.5.232.190:37905 requested a full tablet report, sending...
I20260929 16:30:15.029346 16366 ts_manager.cc:194] Registered new tserver with Master: 3b9f2892f08b442381f7af2fdf9f5d77 (127.5.232.131:43749)
I20260929 16:30:15.030301 16366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:43761
I20260929 16:30:15.039300 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:30:15.055668 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.043754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.5.232.190:37905)
0929 16:30:15.043975 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.043984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.044337 (+ 353us) server_negotiation.cc:407] Connection header received
0929 16:30:15.044469 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.044473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.044533 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.044616 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:15.046069 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.046895 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.048122 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.048506 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.048604 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.050784 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:15.050789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:15.050792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:15.050824 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:15.053260 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.053922 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.053925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.053928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.054047 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.054398 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.054400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.054402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.054699 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:15.054879 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.055166 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.055331 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:30:15.059901 16366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:15.066614 16366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:15.095245 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.079451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:44575 (local address 127.0.0.1:46742)
0929 16:30:15.082812 (+ 3361us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:15.082837 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:15.083006 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:15.084867 (+ 1861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:15.084879 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:15.084930 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:15.085483 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.085501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.088112 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.088121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.091434 (+ 3313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.091456 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.094270 (+ 2814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.094277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.094479 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.094546 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 16:30:15.094682 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 16:30:15.094870 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3195,"thread_start_us":3225,"threads_started":1}
I20260929 16:30:15.095325 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.079857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.5.232.130:44575)
0929 16:30:15.082285 (+ 2428us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.082300 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.083174 (+ 874us) server_negotiation.cc:407] Connection header received
0929 16:30:15.083310 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.083323 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.084219 (+ 896us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.084491 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:15.085756 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.087890 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.092187 (+ 4297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.093789 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.093929 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.094141 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.094773 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.094956 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2200,"thread_start_us":2029,"threads_started":1}
I20260929 16:30:15.099562 16625 tablet_service.cc:1467] Processing CreateTablet for tablet 802f67b6978f4196bfc5b9906906c0f9 (DEFAULT_TABLE table=test-table [id=bd26c543dde9403a83ec9ba0e9296b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:15.100425 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.079265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39141 (local address 127.0.0.1:51112)
0929 16:30:15.080055 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:15.080115 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:15.080344 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:15.092815 (+ 12471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:15.092820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:15.092848 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:15.093261 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.093271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.095383 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.095391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.097492 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.097507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.099463 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.099469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.099694 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.099730 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:30:15.099839 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:30:15.099978 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":1,"thread_start_us":784,"threads_started":1}
I20260929 16:30:15.100483 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.079576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.5.232.129:39141)
0929 16:30:15.092147 (+ 12571us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.092158 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.092186 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:30:15.092298 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.092305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.092591 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.093491 (+ 900us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:15.093730 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.095217 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.097676 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.099214 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.099357 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.099585 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.099908 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.100096 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12380,"thread_start_us":12219,"threads_started":1}
I20260929 16:30:15.101598 16625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802f67b6978f4196bfc5b9906906c0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:15.103942 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.081898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.5.232.131:43749)
0929 16:30:15.094485 (+ 12587us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.094494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.094524 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:30:15.094631 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.094638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.094923 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.095120 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:15.096344 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.097947 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.100398 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.102705 (+ 2307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.102870 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.103114 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.103393 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.103574 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12401,"thread_start_us":12448,"threads_started":1}
I20260929 16:30:15.105293 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.082402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:43749 (local address 127.0.0.1:45598)
0929 16:30:15.094123 (+ 11721us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:15.094145 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:15.094276 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:15.095391 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:15.095396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:15.095419 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:15.096122 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.096133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.098112 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.098118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.100196 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.100213 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.102955 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.102963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.103173 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.103209 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:30:15.103320 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:30:15.104868 (+ 1548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11547,"mutex_wait_us":67,"thread_start_us":11402,"threads_started":1}
I20260929 16:30:15.104885 16491 tablet_service.cc:1467] Processing CreateTablet for tablet 802f67b6978f4196bfc5b9906906c0f9 (DEFAULT_TABLE table=test-table [id=bd26c543dde9403a83ec9ba0e9296b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:15.106340 16491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802f67b6978f4196bfc5b9906906c0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:15.109361 16759 tablet_service.cc:1467] Processing CreateTablet for tablet 802f67b6978f4196bfc5b9906906c0f9 (DEFAULT_TABLE table=test-table [id=bd26c543dde9403a83ec9ba0e9296b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:15.110918 16759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802f67b6978f4196bfc5b9906906c0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:15.117702 16843 tablet_bootstrap.cc:492] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: Bootstrap starting.
I20260929 16:30:15.119867 16844 tablet_bootstrap.cc:492] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: Bootstrap starting.
I20260929 16:30:15.122741 16843 tablet_bootstrap.cc:654] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:15.123047 16844 tablet_bootstrap.cc:654] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:15.124006 16844 log.cc:824] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:15.124053 16843 log.cc:824] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:15.127044 16847 tablet_bootstrap.cc:492] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: Bootstrap starting.
I20260929 16:30:15.127329 16843 tablet_bootstrap.cc:492] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: No bootstrap required, opened a new log
I20260929 16:30:15.127331 16844 tablet_bootstrap.cc:492] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: No bootstrap required, opened a new log
I20260929 16:30:15.127633 16844 ts_tablet_manager.cc:1397] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 16:30:15.127656 16843 ts_tablet_manager.cc:1397] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 16:30:15.131086 16847 tablet_bootstrap.cc:654] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:15.132620 16847 log.cc:824] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:15.135936 16844 raft_consensus.cc:348] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } }
I20260929 16:30:15.136189 16847 tablet_bootstrap.cc:492] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: No bootstrap required, opened a new log
I20260929 16:30:15.136355 16844 raft_consensus.cc:374] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:15.136504 16844 raft_consensus.cc:729] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d2cea82c834ac4b20fb4c10bf20084, State: Initialized, Role: FOLLOWER
I20260929 16:30:15.136515 16847 ts_tablet_manager.cc:1397] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:30:15.137188 16844 consensus_queue.cc:260] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } }
I20260929 16:30:15.138311 16690 heartbeater.cc:503] Master 127.5.232.190:37905 was elected leader, sending a full tablet report...
I20260929 16:30:15.138487 16843 raft_consensus.cc:348] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } }
I20260929 16:30:15.138945 16843 raft_consensus.cc:374] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:15.139114 16843 raft_consensus.cc:729] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20094fd146aa4c208d37f9cf459ef260, State: Initialized, Role: FOLLOWER
I20260929 16:30:15.139940 16844 ts_tablet_manager.cc:1428] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: Time spent starting tablet: real 0.012s user 0.001s sys 0.009s
I20260929 16:30:15.139889 16843 consensus_queue.cc:260] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } }
I20260929 16:30:15.143882 16843 ts_tablet_manager.cc:1428] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260929 16:30:15.147500 16847 raft_consensus.cc:348] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } }
I20260929 16:30:15.147969 16847 raft_consensus.cc:374] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:15.148098 16847 raft_consensus.cc:729] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f2892f08b442381f7af2fdf9f5d77, State: Initialized, Role: FOLLOWER
I20260929 16:30:15.148681 16847 consensus_queue.cc:260] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } }
I20260929 16:30:15.149751 16824 heartbeater.cc:503] Master 127.5.232.190:37905 was elected leader, sending a full tablet report...
I20260929 16:30:15.150486 16847 ts_tablet_manager.cc:1428] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260929 16:30:15.200278 16557 tablet.cc:2369] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:15.221899 16691 tablet.cc:2369] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:15.255790 16825 tablet.cc:2369] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:15.314561 16850 raft_consensus.cc:482] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:15.314925 16850 raft_consensus.cc:504] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } }
I20260929 16:30:15.316723 16850 leader_election.cc:288] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56d2cea82c834ac4b20fb4c10bf20084 (127.5.232.130:44575), 3b9f2892f08b442381f7af2fdf9f5d77 (127.5.232.131:43749)
I20260929 16:30:15.325861 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.317277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:36783 (local address 127.5.232.130:44575)
0929 16:30:15.317504 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.317511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.318562 (+ 1051us) server_negotiation.cc:407] Connection header received
0929 16:30:15.318837 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.318843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.318922 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.318993 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:15.319649 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.320667 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.323364 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.324618 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.324793 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.324873 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.325450 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.325595 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:30:15.325862 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.317330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:41405 (local address 127.5.232.131:43749)
0929 16:30:15.317526 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.317533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.318298 (+ 765us) server_negotiation.cc:407] Connection header received
0929 16:30:15.318575 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.318583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.318655 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.318761 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:15.319327 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.320459 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.323364 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.324638 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.324793 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.324873 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.325450 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.325595 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:30:15.326206 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.317263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:44575 (local address 127.5.232.129:36783)
0929 16:30:15.318426 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:15.318443 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:15.318673 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:15.319152 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:15.319155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:15.319176 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:15.319470 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.319477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.320801 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.320811 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.323141 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.323160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.324946 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.324950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.325177 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.325278 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 16:30:15.325409 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 16:30:15.325701 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":451,"thread_start_us":506,"threads_started":1}
I20260929 16:30:15.326228 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.317251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:43749 (local address 127.5.232.129:41405)
0929 16:30:15.318169 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:15.318195 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:15.318418 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:15.318781 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:15.318785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:15.318811 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:15.319181 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.319191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.320607 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.320611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.323141 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:15.323159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.324934 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.324940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:15.325181 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.325285 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 16:30:15.325421 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 16:30:15.325701 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":241,"thread_start_us":433,"threads_started":1}
I20260929 16:30:15.327346 16645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802f67b6978f4196bfc5b9906906c0f9" candidate_uuid: "20094fd146aa4c208d37f9cf459ef260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d2cea82c834ac4b20fb4c10bf20084" is_pre_election: true
I20260929 16:30:15.327342 16779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802f67b6978f4196bfc5b9906906c0f9" candidate_uuid: "20094fd146aa4c208d37f9cf459ef260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" is_pre_election: true
I20260929 16:30:15.327867 16779 raft_consensus.cc:2454] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20094fd146aa4c208d37f9cf459ef260 in term 0.
I20260929 16:30:15.327867 16645 raft_consensus.cc:2454] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20094fd146aa4c208d37f9cf459ef260 in term 0.
I20260929 16:30:15.328601 16445 leader_election.cc:302] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20094fd146aa4c208d37f9cf459ef260, 56d2cea82c834ac4b20fb4c10bf20084; no voters:
I20260929 16:30:15.329041 16850 raft_consensus.cc:2781] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:15.329197 16850 raft_consensus.cc:482] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:15.329300 16850 raft_consensus.cc:3037] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:15.331509 16850 raft_consensus.cc:504] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } }
I20260929 16:30:15.332175 16850 leader_election.cc:288] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [CANDIDATE]: Term 1 election: Requested vote from peers 56d2cea82c834ac4b20fb4c10bf20084 (127.5.232.130:44575), 3b9f2892f08b442381f7af2fdf9f5d77 (127.5.232.131:43749)
I20260929 16:30:15.332675 16645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802f67b6978f4196bfc5b9906906c0f9" candidate_uuid: "20094fd146aa4c208d37f9cf459ef260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d2cea82c834ac4b20fb4c10bf20084"
I20260929 16:30:15.332715 16779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802f67b6978f4196bfc5b9906906c0f9" candidate_uuid: "20094fd146aa4c208d37f9cf459ef260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f2892f08b442381f7af2fdf9f5d77"
I20260929 16:30:15.332898 16645 raft_consensus.cc:3037] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:15.332916 16779 raft_consensus.cc:3037] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:15.335163 16645 raft_consensus.cc:2454] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20094fd146aa4c208d37f9cf459ef260 in term 1.
I20260929 16:30:15.335168 16779 raft_consensus.cc:2454] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20094fd146aa4c208d37f9cf459ef260 in term 1.
I20260929 16:30:15.335649 16445 leader_election.cc:302] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20094fd146aa4c208d37f9cf459ef260, 3b9f2892f08b442381f7af2fdf9f5d77, 56d2cea82c834ac4b20fb4c10bf20084; no voters:
I20260929 16:30:15.335992 16850 raft_consensus.cc:2781] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:15.336423 16850 raft_consensus.cc:686] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 1 LEADER]: Becoming Leader. State: Replica: 20094fd146aa4c208d37f9cf459ef260, State: Running, Role: LEADER
I20260929 16:30:15.336902 16850 consensus_queue.cc:237] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } }
I20260929 16:30:15.342134 16367 catalog_manager.cc:5495] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20094fd146aa4c208d37f9cf459ef260 (127.5.232.129). New cstate: current_term: 1 leader_uuid: "20094fd146aa4c208d37f9cf459ef260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20094fd146aa4c208d37f9cf459ef260" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 } health_report { overall_health: UNKNOWN } } }
I20260929 16:30:15.388408 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.383270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.5.232.129:39141)
0929 16:30:15.383514 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.383519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.383538 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:30:15.383630 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.383634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.383711 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.383849 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:15.384190 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.384901 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.386273 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.386677 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.386778 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.387245 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:15.387778 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:15.387897 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.388003 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.388083 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 16:30:15.397178 16779 raft_consensus.cc:1261] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [term 1 FOLLOWER]: Refusing update from remote peer 20094fd146aa4c208d37f9cf459ef260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:15.397172 16645 raft_consensus.cc:1261] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [term 1 FOLLOWER]: Refusing update from remote peer 20094fd146aa4c208d37f9cf459ef260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:15.398392 16854 consensus_queue.cc:1035] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f2892f08b442381f7af2fdf9f5d77" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:15.398813 16850 consensus_queue.cc:1035] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56d2cea82c834ac4b20fb4c10bf20084" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 44575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:15.413029 16859 mvcc.cc:205] Tried to move back new op lower bound from 7334704805456367616 to 7334704805223837696. Current Snapshot: MvccSnapshot[applied={T|T < 7334704805456367616}]
I20260929 16:30:15.427809 16367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:15.428118 16367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:15.432441 16367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48b3c91c08a74e47852d061ba4337b01: Sending DeleteTablet for 3 replicas of tablet 802f67b6978f4196bfc5b9906906c0f9
I20260929 16:30:15.433835 16491 tablet_service.cc:1514] Processing DeleteTablet for tablet 802f67b6978f4196bfc5b9906906c0f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:15 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:51112
I20260929 16:30:15.433835 16759 tablet_service.cc:1514] Processing DeleteTablet for tablet 802f67b6978f4196bfc5b9906906c0f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:15 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:45598
I20260929 16:30:15.433816 16625 tablet_service.cc:1514] Processing DeleteTablet for tablet 802f67b6978f4196bfc5b9906906c0f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:15 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:46742
I20260929 16:30:15.437827 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 16428
I20260929 16:30:15.445346 16868 tablet_replica.cc:330] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: stopping tablet replica
I20260929 16:30:15.447135 16868 raft_consensus.cc:2229] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:30:15.448683 16867 tablet_replica.cc:330] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: stopping tablet replica
I20260929 16:30:15.449890 16868 raft_consensus.cc:2258] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:15.450644 16867 raft_consensus.cc:2229] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:15.452097 16867 raft_consensus.cc:2258] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:15.453824 16868 ts_tablet_manager.cc:1905] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:15.456787 16869 tablet_replica.cc:330] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: stopping tablet replica
I20260929 16:30:15.456238 16867 ts_tablet_manager.cc:1905] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:30:15 dist-test-slave-dssx krb5kdc[16322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:30:15.457300 16869 raft_consensus.cc:2229] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:15.464437 16869 raft_consensus.cc:2258] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:15.464516 16868 ts_tablet_manager.cc:1918] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:15.464715 16868 log.cc:1196] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/802f67b6978f4196bfc5b9906906c0f9
I20260929 16:30:15.465969 16868 ts_tablet_manager.cc:1939] T 802f67b6978f4196bfc5b9906906c0f9 P 20094fd146aa4c208d37f9cf459ef260: Deleting consensus metadata
I20260929 16:30:15.467054 16869 ts_tablet_manager.cc:1905] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:15.474574 16869 ts_tablet_manager.cc:1918] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:15.474782 16869 log.cc:1196] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/802f67b6978f4196bfc5b9906906c0f9
I20260929 16:30:15.475318 16869 ts_tablet_manager.cc:1939] T 802f67b6978f4196bfc5b9906906c0f9 P 3b9f2892f08b442381f7af2fdf9f5d77: Deleting consensus metadata
I20260929 16:30:15.476883 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.439689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.5.232.129:39141)
0929 16:30:15.439922 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.439931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.452113 (+ 12182us) server_negotiation.cc:407] Connection header received
0929 16:30:15.452697 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.452706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.452799 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.452889 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:15.455127 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.456371 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.458390 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.459133 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.459241 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.464908 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:15.464924 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:15.464942 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:15.464990 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:15.469242 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.470510 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.470520 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.470523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.470726 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.471504 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.471510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.471513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.475048 (+ 3535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:15.475493 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.476194 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.476489 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:30:15.477751 16351 catalog_manager.cc:4841] TS 3b9f2892f08b442381f7af2fdf9f5d77 (127.5.232.131:43749): tablet 802f67b6978f4196bfc5b9906906c0f9 (table test-table [id=bd26c543dde9403a83ec9ba0e9296b41]) successfully deleted
I20260929 16:30:15.478241 16867 ts_tablet_manager.cc:1918] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:15.478399 16867 log.cc:1196] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/802f67b6978f4196bfc5b9906906c0f9
I20260929 16:30:15.478896 16867 ts_tablet_manager.cc:1939] T 802f67b6978f4196bfc5b9906906c0f9 P 56d2cea82c834ac4b20fb4c10bf20084: Deleting consensus metadata
I20260929 16:30:15.480675 16551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:15.481945 16352 catalog_manager.cc:4841] TS 56d2cea82c834ac4b20fb4c10bf20084 (127.5.232.130:44575): tablet 802f67b6978f4196bfc5b9906906c0f9 (table test-table [id=bd26c543dde9403a83ec9ba0e9296b41]) successfully deleted
I20260929 16:30:15.584074 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 16428
I20260929 16:30:15.584201 16353 catalog_manager.cc:4841] TS 20094fd146aa4c208d37f9cf459ef260 (127.5.232.129:39141): tablet 802f67b6978f4196bfc5b9906906c0f9 (table test-table [id=bd26c543dde9403a83ec9ba0e9296b41]) successfully deleted
I20260929 16:30:15.601832 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 16562
Sep 29 16:30:15 dist-test-slave-dssx krb5kdc[16322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:30:15.616746 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.602649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.5.232.130:44575)
0929 16:30:15.602821 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.602825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.602877 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:30:15.603021 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.603025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.603099 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.603195 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:15.604398 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.605257 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.606757 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.607143 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.607243 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.611281 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:15.611297 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:15.611313 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:15.611351 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:15.614122 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.614767 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.614775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.614777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.614893 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.615211 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.615214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.615215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.615564 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:15.615784 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.616155 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.616365 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:30:15.617465 16685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:15.706763 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 16562
I20260929 16:30:15.726487 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 16696
Sep 29 16:30:15 dist-test-slave-dssx krb5kdc[16322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:30:15.742748 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.727336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.5.232.131:43749)
0929 16:30:15.727513 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.727519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.727661 (+ 142us) server_negotiation.cc:407] Connection header received
0929 16:30:15.727876 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.727884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.727972 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.728095 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:15.729901 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.730651 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.731905 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.732310 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.732430 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.736878 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:15.736895 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:15.736912 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:15.736954 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:15.739915 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.740655 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.740663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.740665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.740823 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.741165 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.741218 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.741221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.741674 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:15.741893 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.742188 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.742420 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:30:15.743449 16819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:15.835585 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 16696
I20260929 16:30:15.853284 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 16335
I20260929 16:30:15.854586 16397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:30:15.962708 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 16335
2026-09-29T16:30:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4441 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:16 dist-test-slave-dssx krb5kdc[16877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:16 dist-test-slave-dssx krb5kdc[16877](info): set up 2 sockets
Sep 29 16:30:16 dist-test-slave-dssx krb5kdc[16877](info): commencing operation
krb5kdc: starting...
W20260929 16:30:17.054329 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:30:17 dist-test-slave-dssx krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.232.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.232.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/rangeradmin_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/rangeradmin_127.5.232.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.232.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.232.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/rangerlookup_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/rangerlookup_127.5.232.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.232.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.232.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/HTTP_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/HTTP_127.5.232.148@KRBTEST.COM.keytab.
I20260929 16:30:17.388729 6050 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_4zdm2/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/postgres -l logfile start
2026-09-29 16:30:20.008 UTC [16907] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:30:20.008 UTC [16907] LOG: listening on IPv4 address "127.5.232.148", port 43109
2026-09-29 16:30:20.015 UTC [16907] LOG: listening on Unix socket "/tmp/.s.PGSQL.43109"
2026-09-29 16:30:20.037 UTC [16910] LOG: database system was shut down at 2026-09-29 16:30:18 UTC
2026-09-29 16:30:20.040 UTC [16907] LOG: database system is ready to accept connections
I20260929 16:30:21.014004 6050 mini_postgres.cc:96] Postgres bound to 43109
2026-09-29 16:30:21.032 UTC [16918] FATAL: database "slave" does not exist
127.5.232.148:43109 - accepting connections
I20260929 16:30:21.032960 6050 mini_ranger.cc:161] Starting Ranger...
I20260929 16:30:21.060405 6050 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:30:21.541189 6050 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:30:21.541352 6050 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-admin
2026-09-29 16:30:21,841 [I] DB FLAVOR :POSTGRES
2026-09-29 16:30:21,842 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:30:21,842 [I] Checking connection..
2026-09-29 16:30:21,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:30:22,110 [I] Checking connection passed.
2026-09-29 16:30:22,110 [I] --------- Verifying version history table ---------
2026-09-29 16:30:22,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:30:22,384 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:30:22,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:30:22,670 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:30:22,670 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:30:22,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:30:22.936 UTC [16982] WARNING: there is no transaction in progress
2026-09-29 16:30:22,943 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:30:22,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:30:23,177 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:30:23,178 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:30:23,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:30:23,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:30:23,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dssx.c.gcp-upstream.internal','N') ;"
2026-09-29 16:30:23,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:30:24,196 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:30:24,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:30:24,467 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:30:24,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:30:24,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:30:24,981 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:30:24,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:30:25.961 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:25.976 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:25.992 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.008 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.024 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.042 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.097 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.104 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.118 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.129 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.140 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.156 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.166 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.173 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.180 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26.187 UTC [17090] WARNING: there is no transaction in progress
2026-09-29 16:30:26,842 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:30:26,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:30:27,094 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:30:27,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:30:27,354 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:30:27,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:30:27,625 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:30:27,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:30:27,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:30:28,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dssx.c.gcp-upstream.internal';"
2026-09-29 16:30:28,359 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:30:28,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:30:28,606 [I] DB_PATCHES have already been applied
I20260929 16:30:28.612205 6050 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-admin:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_4zdm2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:30:28.612356 6050 mini_ranger.cc:193] Using host: 127.5.232.148
I20260929 16:30:28.627732 6050 mini_ranger.cc:227] Ranger admin URL: http://127.5.232.148:41681
Sep 29, 2026 4:30:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:30:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:30:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:30:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:30:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:30:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:30:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:30:29.770298 6050 mini_ranger.cc:160] Time spent starting Ranger: real 8.737s user 0.005s sys 0.037s
Sep 29, 2026 4:30:49 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:30:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 4:30:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:30:49 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:30:50 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:30:51.277904 6050 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.232.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.232.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/rangerkms_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/rangerkms_127.5.232.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.232.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.232.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.232.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/HTTP_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.232.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/HTTP_127.5.232.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:30:51 dist-test-slave-dssx krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699451, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:30:51.474160 6050 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:30:51.500192 6050 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:30:52.360946 6050 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:30:52.361085 6050 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-kms
I20260929 16:30:52.361258 6050 mini_ranger_kms.cc:227] Using postgres at 127.5.232.148:43109
2026-09-29 16:30:52,818 [I] DB FLAVOR :POSTGRES
2026-09-29 16:30:52,818 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:30:52,819 [I] Checking connection
2026-09-29 16:30:52,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:30:53,081 [I] connection success
2026-09-29 16:30:53,081 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:30:53,082 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:30:53,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:30:53,356 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:30:53,357 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:30:53,357 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:30:53,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:43109/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:30:53,649 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:30:54.153950 6050 mini_ranger_kms.cc:310] Created kms service
I20260929 16:30:54.342468 6050 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:30:54.387328 6050 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:30:54 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699451, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:30:54.802366 6050 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:30:54.802714 6050 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-kms:/tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:30:54.802773 6050 mini_ranger_kms.cc:240] Using host: 127.5.232.148
I20260929 16:30:54.824508 6050 mini_ranger_kms.cc:276] Ranger KMS PID: 17270
I20260929 16:30:54.824700 6050 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.232.148:56547
Sep 29, 2026 4:30:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:30:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:30:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:30:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:30:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:30:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:30:57.196229 6050 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.722s user 0.138s sys 0.326s
I20260929 16:30:57.196487 6050 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:30:57.196687 6050 mini_ranger_kms.cc:401] 127.5.232.148:56547/kms/v1/keys
Sep 29 16:30:57 dist-test-slave-dssx krb5kdc[16877](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699457, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.232.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:30:57 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699457, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.232.148@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
Sep 29 16:30:58 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699457, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.232.148@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
Sep 29 16:31:00 dist-test-slave-dssx krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:31:00.487242 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:43557
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:43557
--ranger_config_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:56547/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:00.855526 17321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:00.856006 17321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:00.856169 17321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:00.864634 17321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:00.864799 17321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:00.864903 17321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:00.865000 17321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:00.865080 17321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:00.875675 17321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.232.148:56547/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:43557
--ranger_config_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:43557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:00.877588 17321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:00.879124 17321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:00.892208 17326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:00.892187 17329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:00.892151 17327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:00.892563 17321 server_base.cc:1034] running on GCE node
I20260929 16:31:00.893777 17321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:00.895274 17321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:00.896512 17321 hybrid_clock.cc:648] HybridClock initialized: now 1790699460896467 us; error 43 us; skew 500 ppm
I20260929 16:31:00.896874 17321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:00 dist-test-slave-dssx krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:00.904018 17321 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:31:00.906075 17321 webserver.cc:460] Webserver started at http://127.5.232.190:34295/ using document root <none> and password file <none>
I20260929 16:31:00.906786 17321 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:00.906912 17321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:00.907229 17321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:31:00 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:00.969323 17321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "221730dd80b041e1bccf9bcae87d7427"
format_stamp: "Formatted at 2026-09-29 16:31:00 on dist-test-slave-dssx"
server_key: "ee102855f3b0bc2ccd008316dc1cee35"
server_key_iv: "b6e51ec136a504a9631dde1bfe50639e"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:00.970273 17321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "221730dd80b041e1bccf9bcae87d7427"
format_stamp: "Formatted at 2026-09-29 16:31:00 on dist-test-slave-dssx"
server_key: "ee102855f3b0bc2ccd008316dc1cee35"
server_key_iv: "b6e51ec136a504a9631dde1bfe50639e"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:00.976002 17321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:31:00.996455 17336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:00.998437 17321 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:31:00.998647 17321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "221730dd80b041e1bccf9bcae87d7427"
format_stamp: "Formatted at 2026-09-29 16:31:00 on dist-test-slave-dssx"
server_key: "ee102855f3b0bc2ccd008316dc1cee35"
server_key_iv: "b6e51ec136a504a9631dde1bfe50639e"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:00.998919 17321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:01.015654 17321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:01.020504 17321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:01.020829 17321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:01.041297 17321 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:43557
I20260929 16:31:01.041318 17387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:43557 every 8 connection(s)
I20260929 16:31:01.042717 17321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:01.046452 17388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:01.049016 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 17321
I20260929 16:31:01.049226 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:31:01.050091 6050 external_mini_cluster.cc:1428] Setting key c43a027fd99a9606e72aa93cf636c41f
I20260929 16:31:01.058804 17388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427: Bootstrap starting.
I20260929 16:31:01.061870 17388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:01.062853 17388 log.cc:824] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:31:01 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:31:01.066283 17388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427: No bootstrap required, opened a new log
I20260929 16:31:01.072453 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:01.052584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.5.232.190:43557)
0929 16:31:01.053437 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:01.053458 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:01.053502 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:31:01.055133 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:01.055157 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:01.055557 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:01.055979 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:01.057259 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:01.058597 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:01.059858 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:01.060559 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:01.060740 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:01.065141 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:01.065148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:01.065163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:01.065197 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:01.068282 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:01.068994 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:01.069001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:01.069007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:01.069175 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:01.069637 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:01.069642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:01.069650 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:01.070171 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:01.070489 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:01.070940 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 16:31:01.071347 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":359,"threads_started":1}
I20260929 16:31:01.074272 17388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221730dd80b041e1bccf9bcae87d7427" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43557 } }
I20260929 16:31:01.074728 17388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:01.074828 17388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221730dd80b041e1bccf9bcae87d7427, State: Initialized, Role: FOLLOWER
I20260929 16:31:01.075480 17388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221730dd80b041e1bccf9bcae87d7427" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43557 } }
I20260929 16:31:01.075744 17388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:01.075841 17388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:01.075968 17388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:01.078081 17388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221730dd80b041e1bccf9bcae87d7427" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43557 } }
I20260929 16:31:01.078524 17388 leader_election.cc:302] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221730dd80b041e1bccf9bcae87d7427; no voters:
I20260929 16:31:01.079421 17388 leader_election.cc:288] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:01.079566 17393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:01.080258 17393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [term 1 LEADER]: Becoming Leader. State: Replica: 221730dd80b041e1bccf9bcae87d7427, State: Running, Role: LEADER
I20260929 16:31:01.080817 17393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221730dd80b041e1bccf9bcae87d7427" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43557 } }
I20260929 16:31:01.081663 17388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:01.083732 17395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221730dd80b041e1bccf9bcae87d7427. Latest consensus state: current_term: 1 leader_uuid: "221730dd80b041e1bccf9bcae87d7427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221730dd80b041e1bccf9bcae87d7427" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43557 } } }
I20260929 16:31:01.084173 17395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:01.084010 17394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221730dd80b041e1bccf9bcae87d7427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221730dd80b041e1bccf9bcae87d7427" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43557 } } }
I20260929 16:31:01.084618 17394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:01.090555 17402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:01.095162 17402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:01.095886 17388 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:31:01.115442 17402 catalog_manager.cc:1347] Generated new cluster ID: 92171e8f7b4c49d8bab91e1b60f5ab82
I20260929 16:31:01.115593 17402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:01.136169 17402 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:01.138248 17402 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:01.152138 17402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427: Generated new TSK 0
I20260929 16:31:01.153491 17402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:31:01.925315 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:56547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43557
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:31:02 dist-test-slave-dssx krb5kdc[16877](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699462, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:31:02.364647 17433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:02.365079 17433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:02.365294 17433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:02.377267 17433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:02.377524 17433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:02.377749 17433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:31:02.393877 17433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.232.148:56547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:02.395602 17433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:02.397379 17433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:02.415122 17443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:02.415192 17444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:02.416059 17446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:02.417171 17433 server_base.cc:1034] running on GCE node
I20260929 16:31:02.417986 17433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:02.429348 17433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:02.436928 17433 hybrid_clock.cc:648] HybridClock initialized: now 1790699462436807 us; error 103 us; skew 500 ppm
I20260929 16:31:02.437569 17433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:02 dist-test-slave-dssx krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699462, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:02.441515 17433 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:31:02.445140 17433 webserver.cc:460] Webserver started at http://127.5.232.129:45251/ using document root <none> and password file <none>
I20260929 16:31:02.445979 17433 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:02.446141 17433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:02.446509 17433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:31:02 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699462, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:02.472208 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "364c8344536f473b9318c856b5f9612d"
format_stamp: "Formatted at 2026-09-29 16:31:02 on dist-test-slave-dssx"
server_key: "e8cd69ecf79f8b205e6054449d1969c4"
server_key_iv: "6cabf080b30ffc77c2d9cf9bc7e736dc"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:02.473559 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "364c8344536f473b9318c856b5f9612d"
format_stamp: "Formatted at 2026-09-29 16:31:02 on dist-test-slave-dssx"
server_key: "e8cd69ecf79f8b205e6054449d1969c4"
server_key_iv: "6cabf080b30ffc77c2d9cf9bc7e736dc"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:02.479736 17433 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:31:02.507876 17453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:02.510521 17433 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.000s sys 0.009s
I20260929 16:31:02.510960 17433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "364c8344536f473b9318c856b5f9612d"
format_stamp: "Formatted at 2026-09-29 16:31:02 on dist-test-slave-dssx"
server_key: "e8cd69ecf79f8b205e6054449d1969c4"
server_key_iv: "6cabf080b30ffc77c2d9cf9bc7e736dc"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:02.511317 17433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:02.556915 17433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:02.563506 17433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:02.563897 17433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:02.565212 17433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:02.567306 17433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:02.567420 17433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:02.567617 17433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:02.567694 17433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:31:02 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699462, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:02.796072 17433 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:42939
I20260929 16:31:02.796113 17565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:42939 every 8 connection(s)
I20260929 16:31:02.797930 17433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:02.803449 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 17433
I20260929 16:31:02.803735 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:02.804775 6050 external_mini_cluster.cc:1428] Setting key c2e743c6ddb5a10a744a7e6eb73343ee
Sep 29 16:31:02 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699462, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
I20260929 16:31:02.850211 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:02.801475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:60371 (local address 127.5.232.190:43557)
0929 16:31:02.827262 (+ 25787us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:02.827273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:02.827311 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:31:02.829133 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:02.829143 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:02.829295 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:02.829475 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:02.832007 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:02.833262 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:02.836191 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:02.836886 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:02.837057 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:02.841940 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:02.841948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:02.842036 (+ 88us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:02.842092 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:02.845909 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:02.847036 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:02.847042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:02.847046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:02.847217 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:02.847775 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:02.847780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:02.847784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:02.848175 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:02.848318 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:02.849570 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0929 16:31:02.849823 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25590,"thread_start_us":27928,"threads_started":1}
I20260929 16:31:02.852738 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:02.802389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:43557 (local address 127.5.232.129:60371)
0929 16:31:02.822432 (+ 20043us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:02.822541 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:02.828426 (+ 5885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:02.830460 (+ 2034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:02.830474 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:02.831084 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:02.831766 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:02.831787 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:02.833560 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:02.833566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:02.835936 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:02.835958 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:02.837474 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:02.837483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:02.837824 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:02.838732 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:02.838763 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:02.841633 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:02.846147 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:02.846159 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:02.846181 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:02.846861 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:02.847355 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:02.847361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:02.847364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:02.847640 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:02.848416 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:02.848426 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:02.848906 (+ 480us) client_negotiation.cc:769] Sending connection context
0929 16:31:02.849223 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 16:31:02.851253 (+ 2030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19502,"thread_start_us":22838,"threads_started":1}
I20260929 16:31:02.855226 17566 heartbeater.cc:347] Connected to a master server at 127.5.232.190:43557
I20260929 16:31:02.855623 17566 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:02.856518 17566 heartbeater.cc:511] Master 127.5.232.190:43557 requested a full tablet report, sending...
I20260929 16:31:02.859282 17353 ts_manager.cc:194] Registered new tserver with Master: 364c8344536f473b9318c856b5f9612d (127.5.232.129:42939)
I20260929 16:31:02.861532 17353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:60371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:31:02.880579 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:56547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43557
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:31:03 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699462, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
W20260929 16:31:03.359252 17572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:03.359666 17572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:03.359890 17572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:03.372882 17572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:03.373064 17572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:03.373246 17572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:31:03.390205 17572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.232.148:56547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:03.392464 17572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:03.394593 17572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:03.416342 17580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:03.416345 17581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:03.426466 17584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:03.426817 17572 server_base.cc:1034] running on GCE node
I20260929 16:31:03.427547 17572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:03.428691 17572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:03.429924 17572 hybrid_clock.cc:648] HybridClock initialized: now 1790699463429844 us; error 65 us; skew 500 ppm
I20260929 16:31:03.430269 17572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:03 dist-test-slave-dssx krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699463, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:03.433583 17572 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:31:03.435362 17572 webserver.cc:460] Webserver started at http://127.5.232.130:38893/ using document root <none> and password file <none>
I20260929 16:31:03.436110 17572 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:03.436210 17572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:03.436566 17572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:31:03 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699463, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:03.457705 17572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf"
format_stamp: "Formatted at 2026-09-29 16:31:03 on dist-test-slave-dssx"
server_key: "3dbc8f1559d41af4f019f663d8ce0669"
server_key_iv: "b2e78ab9ae35e0f9348f4b49619a86f2"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:03.458986 17572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf"
format_stamp: "Formatted at 2026-09-29 16:31:03 on dist-test-slave-dssx"
server_key: "3dbc8f1559d41af4f019f663d8ce0669"
server_key_iv: "b2e78ab9ae35e0f9348f4b49619a86f2"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:03.467343 17572 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260929 16:31:03.494114 17596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:03.496359 17572 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.005s
I20260929 16:31:03.496600 17572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf"
format_stamp: "Formatted at 2026-09-29 16:31:03 on dist-test-slave-dssx"
server_key: "3dbc8f1559d41af4f019f663d8ce0669"
server_key_iv: "b2e78ab9ae35e0f9348f4b49619a86f2"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:03.496881 17572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:03.525786 17572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:03.531955 17572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:03.532327 17572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:03.533541 17572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:03.535327 17572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:03.535439 17572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:03.535642 17572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:03.535715 17572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:03.577337 17572 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:38739
I20260929 16:31:03.577394 17708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:38739 every 8 connection(s)
I20260929 16:31:03.578722 17572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:03.581621 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 17572
I20260929 16:31:03.581820 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:03.582515 6050 external_mini_cluster.cc:1428] Setting key 1796a53f73fe30deda33dc49f2e42c43
Sep 29 16:31:03 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699463, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
I20260929 16:31:03.618464 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:03.581610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:55289 (local address 127.5.232.190:43557)
0929 16:31:03.596888 (+ 15278us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:03.596898 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:03.596927 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:31:03.602141 (+ 5214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:03.602149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:03.602246 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:03.602328 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:03.604730 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:03.605948 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:03.607591 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:03.608005 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:03.608147 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:03.611749 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:03.611754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:03.611757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:03.611791 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:03.614587 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:03.615455 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:03.615463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:03.615470 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:03.615593 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:03.616030 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:03.616038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:03.616041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:03.616389 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:03.616545 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:03.617832 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0929 16:31:03.618089 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15122,"thread_start_us":15308,"threads_started":1}
I20260929 16:31:03.619851 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:03.582284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:43557 (local address 127.5.232.130:55289)
0929 16:31:03.595469 (+ 13185us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:03.595545 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:03.598070 (+ 2525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:03.602979 (+ 4909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:03.602989 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:03.603608 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:03.604507 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:03.604539 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:03.606115 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:03.606121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:03.607432 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:03.607446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:03.608242 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:03.608250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:03.608521 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:03.609326 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:03.609356 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:03.611503 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:03.614851 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:03.614859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:03.614874 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:03.615311 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:03.615709 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:03.615713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:03.615717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:03.615914 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:03.616776 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:03.616785 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:03.617257 (+ 472us) client_negotiation.cc:769] Sending connection context
0929 16:31:03.617628 (+ 371us) client_negotiation.cc:240] Negotiation successful
0929 16:31:03.618262 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12806,"thread_start_us":19078,"threads_started":1}
I20260929 16:31:03.623688 17709 heartbeater.cc:347] Connected to a master server at 127.5.232.190:43557
I20260929 16:31:03.624078 17709 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:03.624951 17709 heartbeater.cc:511] Master 127.5.232.190:43557 requested a full tablet report, sending...
I20260929 16:31:03.626792 17353 ts_manager.cc:194] Registered new tserver with Master: dbafa61e2a954484a68b7d3b1f7f3ecf (127.5.232.130:38739)
I20260929 16:31:03.628073 17353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:55289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:31:03.646358 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:56547/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43557
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:31:03.868667 17566 heartbeater.cc:503] Master 127.5.232.190:43557 was elected leader, sending a full tablet report...
W20260929 16:31:04.006500 17715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:04.007292 17715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:04.007488 17715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:04.015025 17715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:04.015152 17715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:04.015324 17715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:31:04.025975 17715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.232.148:56547/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:04.027364 17715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:04.028740 17715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:04.048828 17720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:04.049255 17723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:04.049870 17721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:04.051038 17715 server_base.cc:1034] running on GCE node
I20260929 16:31:04.051752 17715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:04.052881 17715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:04.054113 17715 hybrid_clock.cc:648] HybridClock initialized: now 1790699464054071 us; error 46 us; skew 500 ppm
I20260929 16:31:04.054471 17715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:04 dist-test-slave-dssx krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699464, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:04.058723 17715 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:31:04.060508 17715 webserver.cc:460] Webserver started at http://127.5.232.131:35967/ using document root <none> and password file <none>
I20260929 16:31:04.061331 17715 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:04.061568 17715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:04.061925 17715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:31:04 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699464, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:04.081948 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "4b760e6b6cdf498e91bfe1a8e5893821"
format_stamp: "Formatted at 2026-09-29 16:31:04 on dist-test-slave-dssx"
server_key: "661369820d76c779c08a1d1333f92e9a"
server_key_iv: "21ccd2b206b6f89ed5a27599b86785e9"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:04.083165 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "4b760e6b6cdf498e91bfe1a8e5893821"
format_stamp: "Formatted at 2026-09-29 16:31:04 on dist-test-slave-dssx"
server_key: "661369820d76c779c08a1d1333f92e9a"
server_key_iv: "21ccd2b206b6f89ed5a27599b86785e9"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:04.090938 17715 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 16:31:04.107659 17730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:04.109439 17715 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 16:31:04.109678 17715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "4b760e6b6cdf498e91bfe1a8e5893821"
format_stamp: "Formatted at 2026-09-29 16:31:04 on dist-test-slave-dssx"
server_key: "661369820d76c779c08a1d1333f92e9a"
server_key_iv: "21ccd2b206b6f89ed5a27599b86785e9"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:04.109913 17715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:04.133234 17715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:04.137576 17715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:04.137913 17715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:04.138904 17715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:04.140362 17715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:04.140467 17715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:04.140622 17715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:04.140698 17715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:04.177551 17715 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:45769
I20260929 16:31:04.177595 17842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:45769 every 8 connection(s)
I20260929 16:31:04.178930 17715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:04.181327 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 17715
I20260929 16:31:04.181529 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:04.182042 6050 external_mini_cluster.cc:1428] Setting key 4c3943a8275ced53eaa0373919d304b0
Sep 29 16:31:04 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699464, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:31:04.200848 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.182648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:52531 (local address 127.5.232.190:43557)
0929 16:31:04.183222 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.183230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.184465 (+ 1235us) server_negotiation.cc:407] Connection header received
0929 16:31:04.186540 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.186545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.186651 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.186780 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:04.188742 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.189578 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.191144 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.191608 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.191744 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.195070 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:04.195075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:04.195078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:04.195111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:04.198263 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.198963 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.198967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.198969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.199086 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.199465 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.199468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.199469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.199721 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:04.199829 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.200439 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.200601 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":290,"threads_started":1}
I20260929 16:31:04.201975 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.183372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:43557 (local address 127.5.232.131:52531)
0929 16:31:04.184298 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:04.184349 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:04.186237 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:04.187032 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:04.187041 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:04.187546 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:04.188565 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.188589 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.189734 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.189739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.190991 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.191003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.191835 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.191842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.192101 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.192744 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:04.192767 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:04.194846 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:04.198386 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:04.198394 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:04.198408 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:04.198866 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:04.199180 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:04.199183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:04.199185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:04.199375 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:04.199823 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:04.199829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:04.200144 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 16:31:04.200369 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 16:31:04.200761 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":310,"threads_started":1}
I20260929 16:31:04.203466 17843 heartbeater.cc:347] Connected to a master server at 127.5.232.190:43557
I20260929 16:31:04.203737 17843 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:04.205037 17843 heartbeater.cc:511] Master 127.5.232.190:43557 requested a full tablet report, sending...
I20260929 16:31:04.206558 17353 ts_manager.cc:194] Registered new tserver with Master: 4b760e6b6cdf498e91bfe1a8e5893821 (127.5.232.131:45769)
I20260929 16:31:04.207391 17353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:52531
I20260929 16:31:04.210310 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:31:04.227530 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.214913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.5.232.190:43557)
0929 16:31:04.215076 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.215081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.215603 (+ 522us) server_negotiation.cc:407] Connection header received
0929 16:31:04.215742 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.215746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.215819 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.215901 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:04.217367 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.218198 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.219568 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.219975 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.220073 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.222358 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:04.222362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:04.222366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:04.222397 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:04.224816 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.225459 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.225463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.225465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.225584 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.225900 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.225903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.225905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.226584 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:04.226757 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.227034 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.227199 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:31:04.231418 17353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:04.238622 17353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:31:04.275331 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.251411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:42939 (local address 127.0.0.1:46160)
0929 16:31:04.266285 (+ 14874us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:04.266319 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:04.266454 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:04.267172 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:04.267178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:04.267217 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:04.267837 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.267847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.269957 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.269963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.272579 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.272591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.274271 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.274278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.274464 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.274529 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 16:31:04.274663 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:31:04.274871 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14752,"mutex_wait_us":25,"thread_start_us":12355,"threads_started":1}
I20260929 16:31:04.276134 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.252173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.5.232.129:42939)
0929 16:31:04.265476 (+ 13303us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.265490 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.266595 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 16:31:04.266700 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.266706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.266931 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.267611 (+ 680us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:04.268214 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.269804 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.272739 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.274068 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.274215 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.274471 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.275593 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.275778 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13039,"thread_start_us":14522,"threads_started":1}
I20260929 16:31:04.276221 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.266725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:38739 (local address 127.0.0.1:38544)
0929 16:31:04.267372 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:04.267394 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:04.267511 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:04.268305 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:04.268308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:04.268338 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:04.268599 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.268606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.270042 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.270046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.273146 (+ 3100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.273160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.275420 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.275426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.275589 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.275620 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:31:04.275704 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:31:04.275825 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":140,"thread_start_us":342,"threads_started":1}
I20260929 16:31:04.276438 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.252077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.5.232.131:45769)
0929 16:31:04.266010 (+ 13933us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.266020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.266061 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:31:04.266174 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.266183 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.266493 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.266688 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:04.268009 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.269673 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.273321 (+ 3648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.274758 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.274883 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.275116 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.275935 (+ 819us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.276090 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13739,"thread_start_us":13585,"threads_started":1}
I20260929 16:31:04.276656 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.251301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:45769 (local address 127.0.0.1:55714)
0929 16:31:04.264341 (+ 13040us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:04.264410 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:04.264674 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:04.267010 (+ 2336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:04.267021 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:04.267071 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:04.267781 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.267796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.269803 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.269809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.273152 (+ 3343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.273160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.275275 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.275281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.275750 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.275781 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:31:04.275889 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:31:04.276031 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12650,"mutex_wait_us":193,"thread_start_us":14706,"threads_started":1}
I20260929 16:31:04.276942 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.265302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.5.232.130:38739)
0929 16:31:04.266661 (+ 1359us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.266669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.267566 (+ 897us) server_negotiation.cc:407] Connection header received
0929 16:31:04.267664 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.267670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.267902 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.268085 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:04.268752 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.269900 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.273486 (+ 3586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.274942 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.275077 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.275253 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.276444 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.276625 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":1087,"threads_started":1}
I20260929 16:31:04.280390 17778 tablet_service.cc:1467] Processing CreateTablet for tablet 7537e58999a4470baf5158daf4408591 (DEFAULT_TABLE table=test-table [id=615c46b654a147f3abd54c2748f89be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:04.280414 17501 tablet_service.cc:1467] Processing CreateTablet for tablet 7537e58999a4470baf5158daf4408591 (DEFAULT_TABLE table=test-table [id=615c46b654a147f3abd54c2748f89be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:04.280414 17644 tablet_service.cc:1467] Processing CreateTablet for tablet 7537e58999a4470baf5158daf4408591 (DEFAULT_TABLE table=test-table [id=615c46b654a147f3abd54c2748f89be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:04.281940 17778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7537e58999a4470baf5158daf4408591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:04.282282 17501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7537e58999a4470baf5158daf4408591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:04.282282 17644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7537e58999a4470baf5158daf4408591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:04.294759 17862 tablet_bootstrap.cc:492] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: Bootstrap starting.
I20260929 16:31:04.297008 17863 tablet_bootstrap.cc:492] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: Bootstrap starting.
I20260929 16:31:04.300704 17862 tablet_bootstrap.cc:654] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:04.301034 17863 tablet_bootstrap.cc:654] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:04.301774 17864 tablet_bootstrap.cc:492] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: Bootstrap starting.
I20260929 16:31:04.302171 17862 log.cc:824] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:04.302309 17863 log.cc:824] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:04.305568 17862 tablet_bootstrap.cc:492] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: No bootstrap required, opened a new log
I20260929 16:31:04.305753 17863 tablet_bootstrap.cc:492] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: No bootstrap required, opened a new log
I20260929 16:31:04.305801 17864 tablet_bootstrap.cc:654] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:04.305878 17862 ts_tablet_manager.cc:1397] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 16:31:04.306030 17863 ts_tablet_manager.cc:1397] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260929 16:31:04.307849 17864 log.cc:824] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:04.310984 17864 tablet_bootstrap.cc:492] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: No bootstrap required, opened a new log
I20260929 16:31:04.311254 17864 ts_tablet_manager.cc:1397] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:31:04.314705 17862 raft_consensus.cc:348] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } }
I20260929 16:31:04.315233 17862 raft_consensus.cc:374] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:04.315335 17862 raft_consensus.cc:729] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b760e6b6cdf498e91bfe1a8e5893821, State: Initialized, Role: FOLLOWER
I20260929 16:31:04.315963 17862 consensus_queue.cc:260] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } }
I20260929 16:31:04.316435 17863 raft_consensus.cc:348] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } }
I20260929 16:31:04.316854 17863 raft_consensus.cc:374] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:04.316941 17863 raft_consensus.cc:729] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbafa61e2a954484a68b7d3b1f7f3ecf, State: Initialized, Role: FOLLOWER
I20260929 16:31:04.317592 17863 consensus_queue.cc:260] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } }
I20260929 16:31:04.317770 17862 ts_tablet_manager.cc:1428] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: Time spent starting tablet: real 0.012s user 0.001s sys 0.011s
I20260929 16:31:04.318607 17843 heartbeater.cc:503] Master 127.5.232.190:43557 was elected leader, sending a full tablet report...
I20260929 16:31:04.319375 17709 heartbeater.cc:503] Master 127.5.232.190:43557 was elected leader, sending a full tablet report...
I20260929 16:31:04.320096 17863 ts_tablet_manager.cc:1428] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 16:31:04.322885 17864 raft_consensus.cc:348] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } }
I20260929 16:31:04.323387 17864 raft_consensus.cc:374] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:04.323511 17864 raft_consensus.cc:729] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364c8344536f473b9318c856b5f9612d, State: Initialized, Role: FOLLOWER
I20260929 16:31:04.324046 17864 consensus_queue.cc:260] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } }
I20260929 16:31:04.325701 17864 ts_tablet_manager.cc:1428] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260929 16:31:04.331225 17710 tablet.cc:2369] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:31:04.331822 17567 tablet.cc:2369] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:04.421538 17868 raft_consensus.cc:482] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:04.421902 17868 raft_consensus.cc:504] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } }
I20260929 16:31:04.423863 17868 leader_election.cc:288] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 364c8344536f473b9318c856b5f9612d (127.5.232.129:42939), dbafa61e2a954484a68b7d3b1f7f3ecf (127.5.232.130:38739)
I20260929 16:31:04.429580 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.424383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:49833 (local address 127.5.232.129:42939)
0929 16:31:04.424636 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.424644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.424667 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:31:04.424766 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.424775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.424865 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.424960 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:04.425417 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.426141 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.427852 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.428641 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.428826 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.428900 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.429170 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.429300 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 16:31:04.429593 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.424121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:42939 (local address 127.5.232.131:49833)
0929 16:31:04.424263 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:04.424279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:04.424377 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:04.425053 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:04.425058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:04.425088 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:04.425249 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.425256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.426317 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.426330 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.427678 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.427692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.428805 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.428811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.428994 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.429017 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:31:04.429134 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 16:31:04.429271 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":3}
I20260929 16:31:04.430399 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.424502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:38739 (local address 127.5.232.131:43847)
0929 16:31:04.425104 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:04.425117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:04.425260 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:04.425611 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:04.425615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:04.425637 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:04.425970 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.425977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.427024 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.427029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.428370 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:04.428382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.429540 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.429546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:04.429749 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.429797 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 16:31:04.429891 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:31:04.430036 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":2,"thread_start_us":319,"threads_started":1}
I20260929 16:31:04.430398 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.424657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:43847 (local address 127.5.232.130:38739)
0929 16:31:04.424893 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.424900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.425212 (+ 312us) server_negotiation.cc:407] Connection header received
0929 16:31:04.425431 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.425436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.425496 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.425596 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:04.426125 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.426848 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.428529 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.429307 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.429514 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.429588 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.429997 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.430136 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 16:31:04.430761 17521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7537e58999a4470baf5158daf4408591" candidate_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364c8344536f473b9318c856b5f9612d" is_pre_election: true
W20260929 16:31:04.431497 17844 tablet.cc:2369] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:31:04.431880 17521 raft_consensus.cc:2454] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b760e6b6cdf498e91bfe1a8e5893821 in term 0.
I20260929 16:31:04.431769 17664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7537e58999a4470baf5158daf4408591" candidate_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" is_pre_election: true
I20260929 16:31:04.432312 17664 raft_consensus.cc:2454] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b760e6b6cdf498e91bfe1a8e5893821 in term 0.
I20260929 16:31:04.432631 17732 leader_election.cc:302] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c8344536f473b9318c856b5f9612d, 4b760e6b6cdf498e91bfe1a8e5893821; no voters:
I20260929 16:31:04.433012 17868 raft_consensus.cc:2781] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:04.433161 17868 raft_consensus.cc:482] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:04.433266 17868 raft_consensus.cc:3037] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:04.435516 17868 raft_consensus.cc:504] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } }
I20260929 16:31:04.436093 17868 leader_election.cc:288] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [CANDIDATE]: Term 1 election: Requested vote from peers 364c8344536f473b9318c856b5f9612d (127.5.232.129:42939), dbafa61e2a954484a68b7d3b1f7f3ecf (127.5.232.130:38739)
I20260929 16:31:04.436658 17521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7537e58999a4470baf5158daf4408591" candidate_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364c8344536f473b9318c856b5f9612d"
I20260929 16:31:04.436674 17664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7537e58999a4470baf5158daf4408591" candidate_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf"
I20260929 16:31:04.436880 17664 raft_consensus.cc:3037] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:04.436872 17521 raft_consensus.cc:3037] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:04.439152 17664 raft_consensus.cc:2454] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b760e6b6cdf498e91bfe1a8e5893821 in term 1.
I20260929 16:31:04.439219 17521 raft_consensus.cc:2454] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b760e6b6cdf498e91bfe1a8e5893821 in term 1.
I20260929 16:31:04.439612 17732 leader_election.cc:302] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364c8344536f473b9318c856b5f9612d, 4b760e6b6cdf498e91bfe1a8e5893821; no voters:
I20260929 16:31:04.439899 17868 raft_consensus.cc:2781] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:04.440333 17868 raft_consensus.cc:686] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 1 LEADER]: Becoming Leader. State: Replica: 4b760e6b6cdf498e91bfe1a8e5893821, State: Running, Role: LEADER
I20260929 16:31:04.440809 17868 consensus_queue.cc:237] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } }
I20260929 16:31:04.445842 17353 catalog_manager.cc:5495] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b760e6b6cdf498e91bfe1a8e5893821 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b760e6b6cdf498e91bfe1a8e5893821" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:04.472878 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.467629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.5.232.131:45769)
0929 16:31:04.467811 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.467816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.467834 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:31:04.467972 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.467976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.468057 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.468185 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:04.468505 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.469204 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.470691 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.471087 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.471183 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.471709 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:04.472254 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:04.472379 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.472476 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.472550 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:31:04.481552 17664 raft_consensus.cc:1261] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [term 1 FOLLOWER]: Refusing update from remote peer 4b760e6b6cdf498e91bfe1a8e5893821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:04.481585 17521 raft_consensus.cc:1261] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [term 1 FOLLOWER]: Refusing update from remote peer 4b760e6b6cdf498e91bfe1a8e5893821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:04.482724 17872 consensus_queue.cc:1035] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364c8344536f473b9318c856b5f9612d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:04.483084 17868 consensus_queue.cc:1035] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbafa61e2a954484a68b7d3b1f7f3ecf" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:04.496579 17875 mvcc.cc:205] Tried to move back new op lower bound from 7334705006505562112 to 7334705006353842176. Current Snapshot: MvccSnapshot[applied={T|T < 7334705006505562112}]
I20260929 16:31:04.496927 17877 mvcc.cc:205] Tried to move back new op lower bound from 7334705006505562112 to 7334705006353842176. Current Snapshot: MvccSnapshot[applied={T|T < 7334705006505562112}]
I20260929 16:31:04.516505 17352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:04.516846 17352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:04.520773 17352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 221730dd80b041e1bccf9bcae87d7427: Sending DeleteTablet for 3 replicas of tablet 7537e58999a4470baf5158daf4408591
I20260929 16:31:04.522202 17644 tablet_service.cc:1514] Processing DeleteTablet for tablet 7537e58999a4470baf5158daf4408591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:04 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:38544
I20260929 16:31:04.522706 17778 tablet_service.cc:1514] Processing DeleteTablet for tablet 7537e58999a4470baf5158daf4408591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:04 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:55714
I20260929 16:31:04.523602 17886 tablet_replica.cc:330] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: stopping tablet replica
I20260929 16:31:04.523691 17501 tablet_service.cc:1514] Processing DeleteTablet for tablet 7537e58999a4470baf5158daf4408591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:04 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:46160
I20260929 16:31:04.524158 17885 tablet_replica.cc:330] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: stopping tablet replica
I20260929 16:31:04.524590 17885 raft_consensus.cc:2229] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:04.524839 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 17433
I20260929 16:31:04.525295 17885 raft_consensus.cc:2258] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:04.526028 17886 raft_consensus.cc:2229] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:04.526559 17886 raft_consensus.cc:2258] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:04.528744 17885 ts_tablet_manager.cc:1905] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:04.534063 17886 ts_tablet_manager.cc:1905] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:04.537339 17885 ts_tablet_manager.cc:1918] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:04.537575 17885 log.cc:1196] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/7537e58999a4470baf5158daf4408591
I20260929 16:31:04.538195 17885 ts_tablet_manager.cc:1939] T 7537e58999a4470baf5158daf4408591 P 4b760e6b6cdf498e91bfe1a8e5893821: Deleting consensus metadata
I20260929 16:31:04.540223 17337 catalog_manager.cc:4841] TS 4b760e6b6cdf498e91bfe1a8e5893821 (127.5.232.131:45769): tablet 7537e58999a4470baf5158daf4408591 (table test-table [id=615c46b654a147f3abd54c2748f89be0]) successfully deleted
I20260929 16:31:04.540622 17887 tablet_replica.cc:330] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: stopping tablet replica
I20260929 16:31:04.541352 17887 raft_consensus.cc:2229] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:04.542007 17887 raft_consensus.cc:2258] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:04.544515 17887 ts_tablet_manager.cc:1905] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:04.548085 17886 ts_tablet_manager.cc:1918] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:04.548316 17886 log.cc:1196] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/7537e58999a4470baf5158daf4408591
I20260929 16:31:04.548900 17886 ts_tablet_manager.cc:1939] T 7537e58999a4470baf5158daf4408591 P dbafa61e2a954484a68b7d3b1f7f3ecf: Deleting consensus metadata
Sep 29 16:31:04 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:31:04.551096 17338 catalog_manager.cc:4841] TS dbafa61e2a954484a68b7d3b1f7f3ecf (127.5.232.130:38739): tablet 7537e58999a4470baf5158daf4408591 (table test-table [id=615c46b654a147f3abd54c2748f89be0]) successfully deleted
I20260929 16:31:04.552536 17887 ts_tablet_manager.cc:1918] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:04.552758 17887 log.cc:1196] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/7537e58999a4470baf5158daf4408591
I20260929 16:31:04.553346 17887 ts_tablet_manager.cc:1939] T 7537e58999a4470baf5158daf4408591 P 364c8344536f473b9318c856b5f9612d: Deleting consensus metadata
I20260929 16:31:04.555464 17338 catalog_manager.cc:4841] TS 364c8344536f473b9318c856b5f9612d (127.5.232.129:42939): tablet 7537e58999a4470baf5158daf4408591 (table test-table [id=615c46b654a147f3abd54c2748f89be0]) successfully deleted
I20260929 16:31:04.559028 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.527221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.5.232.129:42939)
0929 16:31:04.527452 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.527459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.538190 (+ 10731us) server_negotiation.cc:407] Connection header received
0929 16:31:04.538676 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.538686 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.538809 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.538918 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:04.541308 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.542533 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.544650 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.545324 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.545464 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.552126 (+ 6662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:04.552141 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:04.552156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:04.552195 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:04.555818 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.556637 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.556650 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.556658 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.556798 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.557222 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.557228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.557231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.557694 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:04.558002 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.558353 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.558612 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:31:04.559794 17561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:04.667245 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 17433
I20260929 16:31:04.684790 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 17572
Sep 29 16:31:04 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:31:04.703608 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.685626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.5.232.130:38739)
0929 16:31:04.685852 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.685856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.685944 (+ 88us) server_negotiation.cc:407] Connection header received
0929 16:31:04.686163 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.686167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.686242 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.686344 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:04.688068 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.688812 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.690871 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.691255 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.691361 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.697514 (+ 6153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:04.697537 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:04.697562 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:04.697618 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:04.700611 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.701358 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.701366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.701368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.701568 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.701979 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.701984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.701987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.702367 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:04.702658 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.703018 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.703246 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:31:04.704384 17704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:04.795059 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 17572
I20260929 16:31:04.812593 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 17715
Sep 29 16:31:04 dist-test-slave-dssx krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:31:04.831121 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:04.813563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.5.232.131:45769)
0929 16:31:04.813749 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:04.813753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:04.813772 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:31:04.813873 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:04.813877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:04.813941 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:04.814037 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:04.815951 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.816745 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.818789 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:04.819252 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:04.819361 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:04.824872 (+ 5511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:04.824887 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:04.824902 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:04.824952 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:04.828092 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.828878 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.828886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.828888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.829038 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:04.829470 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:04.829475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:04.829477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:04.829858 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:04.830184 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:04.830511 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 16:31:04.830734 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:31:04.831890 17838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:04.955499 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 17715
I20260929 16:31:04.975020 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 17321
I20260929 16:31:04.976234 17383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:05.088677 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 17321
I20260929 16:31:05.110996 6050 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:31:05.446960 6050 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:31:05.447162 6050 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:31:05.792984 6050 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:31:05.793 UTC [16907] LOG: received smart shutdown request
2026-09-29 16:31:05.796 UTC [16907] LOG: background worker "logical replication launcher" (PID 16916) exited with exit code 1
2026-09-29 16:31:05.797 UTC [16911] LOG: shutting down
2026-09-29 16:31:05.954 UTC [16907] LOG: database system is shut down
2026-09-29T16:31:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50109 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:06 dist-test-slave-dssx krb5kdc[17901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:06 dist-test-slave-dssx krb5kdc[17901](info): set up 2 sockets
Sep 29 16:31:06 dist-test-slave-dssx krb5kdc[17901](info): commencing operation
krb5kdc: starting...
W20260929 16:31:07.185691 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:31:07 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:31:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.232.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.232.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/rangeradmin_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/rangeradmin_127.5.232.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.232.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.232.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/rangerlookup_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/rangerlookup_127.5.232.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.232.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.232.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/HTTP_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/HTTP_127.5.232.148@KRBTEST.COM.keytab.
I20260929 16:31:07.504756 6050 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_4zdm2/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/postgres -l logfile start
2026-09-29 16:31:09.949 UTC [17931] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:31:09.949 UTC [17931] LOG: listening on IPv4 address "127.5.232.148", port 47755
2026-09-29 16:31:09.952 UTC [17931] LOG: listening on Unix socket "/tmp/.s.PGSQL.47755"
2026-09-29 16:31:09.974 UTC [17934] LOG: database system was shut down at 2026-09-29 16:31:08 UTC
2026-09-29 16:31:09.978 UTC [17931] LOG: database system is ready to accept connections
I20260929 16:31:10.945413 6050 mini_postgres.cc:96] Postgres bound to 47755
2026-09-29 16:31:10.962 UTC [17942] FATAL: database "slave" does not exist
127.5.232.148:47755 - accepting connections
I20260929 16:31:10.962801 6050 mini_ranger.cc:161] Starting Ranger...
I20260929 16:31:10.987980 6050 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:31:11.482221 6050 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:31:11.482324 6050 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-admin
2026-09-29 16:31:11,776 [I] DB FLAVOR :POSTGRES
2026-09-29 16:31:11,777 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:31:11,777 [I] Checking connection..
2026-09-29 16:31:11,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:31:12,030 [I] Checking connection passed.
2026-09-29 16:31:12,030 [I] --------- Verifying version history table ---------
2026-09-29 16:31:12,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:31:12,264 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:31:12,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:31:12,513 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:31:12,513 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:31:12,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:31:12.772 UTC [18006] WARNING: there is no transaction in progress
2026-09-29 16:31:12,780 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:31:12,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:31:13,022 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:31:13,022 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:31:13,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:31:13,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:31:13,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dssx.c.gcp-upstream.internal','N') ;"
2026-09-29 16:31:13,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:31:14,032 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:31:14,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:31:14,274 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:31:14,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:31:14,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:31:14,768 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:31:14,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:31:15.714 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.732 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.749 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.764 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.780 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.799 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.856 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.863 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.879 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.889 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.899 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.912 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.922 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.929 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.935 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:15.942 UTC [18114] WARNING: there is no transaction in progress
2026-09-29 16:31:16,573 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:31:16,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:31:16,804 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:31:16,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:31:17,067 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:31:17,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:31:17,316 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:31:17,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:31:17,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:31:17,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dssx.c.gcp-upstream.internal';"
2026-09-29 16:31:18,028 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:31:18,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:31:18,290 [I] DB_PATCHES have already been applied
I20260929 16:31:18.295987 6050 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-admin:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_4zdm2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:31:18.296085 6050 mini_ranger.cc:193] Using host: 127.5.232.148
I20260929 16:31:18.311092 6050 mini_ranger.cc:227] Ranger admin URL: http://127.5.232.148:57781
Sep 29, 2026 4:31:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:31:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:31:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:31:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:31:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:31:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:31:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:31:19.434401 6050 mini_ranger.cc:160] Time spent starting Ranger: real 8.472s user 0.002s sys 0.037s
Sep 29, 2026 4:31:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:31:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 4:31:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:31:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:31:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:31:42.303571 6050 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.232.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.232.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/rangerkms_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.232.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/rangerkms_127.5.232.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.232.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.232.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.232.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/HTTP_127.5.232.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.232.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/HTTP_127.5.232.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:31:42 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:31:42.473026 6050 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:31:42.494649 6050 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:31:43.332291 6050 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:31:43.332417 6050 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-kms
I20260929 16:31:43.332463 6050 mini_ranger_kms.cc:227] Using postgres at 127.5.232.148:47755
2026-09-29 16:31:43,697 [I] DB FLAVOR :POSTGRES
2026-09-29 16:31:43,698 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:31:43,698 [I] Checking connection
2026-09-29 16:31:43,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:31:43,978 [I] connection success
2026-09-29 16:31:43,978 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:31:43,978 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:31:43,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:31:44,227 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:31:44,228 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:31:44,228 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:31:44,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.232.148:47755/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:31:44,489 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:31:44.985962 6050 mini_ranger_kms.cc:310] Created kms service
I20260929 16:31:45.182740 6050 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:31:45.229031 6050 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:31:45 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:45.685894 6050 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:31:45.686242 6050 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-kms:/tmp/dist-test-task_4zdm2/build/asan/bin/postgresql.jar:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_4zdm2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:31:45.686290 6050 mini_ranger_kms.cc:240] Using host: 127.5.232.148
I20260929 16:31:45.704932 6050 mini_ranger_kms.cc:276] Ranger KMS PID: 18295
I20260929 16:31:45.705067 6050 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.232.148:48541
Sep 29, 2026 4:31:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:31:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:31:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:31:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:31:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:31:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 16:31:47 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699507, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.232.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:47.942152 6050 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.469s user 0.075s sys 0.300s
I20260929 16:31:47.942417 6050 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:31:47.942615 6050 mini_ranger_kms.cc:401] 127.5.232.148:48541/kms/v1/keys
Sep 29 16:31:48 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699507, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.232.148@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
Sep 29 16:31:48 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699507, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.232.148@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
Sep 29 16:31:50 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:31:50.978101 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46583
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:46583
--ranger_config_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:48541/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:51.322257 18345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:51.322610 18345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:51.322772 18345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:51.330185 18345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:51.330351 18345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:51.330462 18345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:51.330569 18345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:51.330660 18345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:51.340512 18345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.232.148:48541/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:46583
--ranger_config_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:51.341925 18345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:51.343356 18345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:51.355109 18350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:51.355077 18351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:51.355157 18353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:51.356323 18345 server_base.cc:1034] running on GCE node
I20260929 16:31:51.357060 18345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:51.358582 18345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:51.359752 18345 hybrid_clock.cc:648] HybridClock initialized: now 1790699511359710 us; error 42 us; skew 500 ppm
I20260929 16:31:51.360124 18345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:51 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699511, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:51.367050 18345 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:31:51.368618 18345 webserver.cc:460] Webserver started at http://127.5.232.190:34423/ using document root <none> and password file <none>
I20260929 16:31:51.369315 18345 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:51.369480 18345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:51.369789 18345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:31:51 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699511, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:51.428470 18345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "920f11fb5a5a4f088aa9c7166edb17f4"
format_stamp: "Formatted at 2026-09-29 16:31:51 on dist-test-slave-dssx"
server_key: "291961f0c3b0d2ae1394c25dd8d2412d"
server_key_iv: "b4cb43f5da5560d83768b3bd2aec05ce"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:51.429611 18345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "920f11fb5a5a4f088aa9c7166edb17f4"
format_stamp: "Formatted at 2026-09-29 16:31:51 on dist-test-slave-dssx"
server_key: "291961f0c3b0d2ae1394c25dd8d2412d"
server_key_iv: "b4cb43f5da5560d83768b3bd2aec05ce"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:51.436198 18345 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260929 16:31:51.458429 18360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:51.460075 18345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 16:31:51.460284 18345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "920f11fb5a5a4f088aa9c7166edb17f4"
format_stamp: "Formatted at 2026-09-29 16:31:51 on dist-test-slave-dssx"
server_key: "291961f0c3b0d2ae1394c25dd8d2412d"
server_key_iv: "b4cb43f5da5560d83768b3bd2aec05ce"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:51.460526 18345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:51.489980 18345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:51.494388 18345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:51.494730 18345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:51.514006 18345 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:46583
I20260929 16:31:51.514014 18411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:46583 every 8 connection(s)
I20260929 16:31:51.515465 18345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:51.519354 18412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:51.523074 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 18345
I20260929 16:31:51.523294 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:31:51.524106 6050 external_mini_cluster.cc:1428] Setting key 03334bdae99af88439bee877f2f86b07
I20260929 16:31:51.532294 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Bootstrap starting.
I20260929 16:31:51.535521 18412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:51.536532 18412 log.cc:824] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:31:51 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:31:51.539759 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: No bootstrap required, opened a new log
I20260929 16:31:51.547322 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:51.526327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.5.232.190:46583)
0929 16:31:51.527093 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:51.527105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:51.527157 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:31:51.528849 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:51.528871 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:51.529294 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:51.529802 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:51.531163 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.532489 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.533884 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:51.534569 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:51.534802 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:51.538765 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:51.538786 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:51.538804 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:51.538860 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:51.542775 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:51.543466 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:51.543475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:51.543483 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:51.543694 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:51.544031 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:51.544036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:51.544039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:51.544751 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:51.545064 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:51.545450 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 16:31:51.545909 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":350,"threads_started":1}
I20260929 16:31:51.550726 18412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } }
I20260929 16:31:51.551074 18412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:51.551144 18412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920f11fb5a5a4f088aa9c7166edb17f4, State: Initialized, Role: FOLLOWER
I20260929 16:31:51.551635 18412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } }
I20260929 16:31:51.551841 18412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:51.551937 18412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:51.552054 18412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:51.554195 18412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } }
I20260929 16:31:51.554641 18412 leader_election.cc:302] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 920f11fb5a5a4f088aa9c7166edb17f4; no voters:
I20260929 16:31:51.555222 18412 leader_election.cc:288] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:31:51.555380 18417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:51.556102 18417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 1 LEADER]: Becoming Leader. State: Replica: 920f11fb5a5a4f088aa9c7166edb17f4, State: Running, Role: LEADER
I20260929 16:31:51.556650 18417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } }
I20260929 16:31:51.557428 18412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:51.559729 18418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } } }
I20260929 16:31:51.560135 18418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:51.560746 18419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 920f11fb5a5a4f088aa9c7166edb17f4. Latest consensus state: current_term: 1 leader_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } } }
I20260929 16:31:51.561048 18419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:51.562455 18425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:51.567782 18425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:51.573446 18412 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:31:51.587924 18425 catalog_manager.cc:1347] Generated new cluster ID: 84737e06063a45ac990661e4defc576f
I20260929 16:31:51.588117 18425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:51.627099 18425 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:31:51.628609 18425 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:51.638202 18425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Generated new TSK 0
I20260929 16:31:51.639339 18425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:31:52.423626 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:48541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46583
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:31:52 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:31:52.855089 18457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:52.855500 18457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:52.855712 18457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:52.867442 18457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:52.867648 18457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:52.867844 18457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:31:52.881471 18457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.232.148:48541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:52.883355 18457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:52.885190 18457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:52.923177 18468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:52.925525 18467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:52.931216 18457 server_base.cc:1034] running on GCE node
W20260929 16:31:52.937417 18471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:52.938408 18457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:52.939775 18457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:52.941109 18457 hybrid_clock.cc:648] HybridClock initialized: now 1790699512940993 us; error 106 us; skew 500 ppm
I20260929 16:31:52.941572 18457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:52 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:52.948468 18457 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:31:52.970398 18457 webserver.cc:460] Webserver started at http://127.5.232.129:42429/ using document root <none> and password file <none>
I20260929 16:31:52.971309 18457 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:52.971487 18457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:52.971837 18457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:31:52 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:52.989840 18457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "a9f97542d118472c834ea3189d30efc5"
format_stamp: "Formatted at 2026-09-29 16:31:52 on dist-test-slave-dssx"
server_key: "2355bf17b370f30f467ff5ec09806f29"
server_key_iv: "02ae7c8d9e49fed2c5a87640a39c909a"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:52.990767 18457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "a9f97542d118472c834ea3189d30efc5"
format_stamp: "Formatted at 2026-09-29 16:31:52 on dist-test-slave-dssx"
server_key: "2355bf17b370f30f467ff5ec09806f29"
server_key_iv: "02ae7c8d9e49fed2c5a87640a39c909a"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:52.998907 18457 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260929 16:31:53.038209 18477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:53.043130 18457 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.011s sys 0.007s
I20260929 16:31:53.043455 18457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "a9f97542d118472c834ea3189d30efc5"
format_stamp: "Formatted at 2026-09-29 16:31:52 on dist-test-slave-dssx"
server_key: "2355bf17b370f30f467ff5ec09806f29"
server_key_iv: "02ae7c8d9e49fed2c5a87640a39c909a"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:53.043839 18457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:53.085170 18457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:53.091002 18457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:53.091418 18457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:53.094724 18457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:53.096709 18457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:53.096848 18457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:31:53.097008 18457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:53.097107 18457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:31:53 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:53.221818 18457 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:36517
I20260929 16:31:53.221861 18589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:36517 every 8 connection(s)
I20260929 16:31:53.223762 18457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:53.233251 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 18457
I20260929 16:31:53.233490 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:31:53.234273 6050 external_mini_cluster.cc:1428] Setting key 097f953d995ad9256c55dfc623aa4503
Sep 29 16:31:53 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:31:53.266219 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:53.234401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:46913 (local address 127.5.232.190:46583)
0929 16:31:53.236107 (+ 1706us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:53.236115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:53.243073 (+ 6958us) server_negotiation.cc:407] Connection header received
0929 16:31:53.244643 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:53.244650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:53.244756 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:53.244856 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:53.247336 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:53.248495 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:53.251021 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:53.251674 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:53.251784 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:53.257577 (+ 5793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:53.257586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:53.257594 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:53.257655 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:53.261707 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:53.262753 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:53.262761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:53.262768 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:53.262995 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:53.263586 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:53.263594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:53.263599 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:53.264090 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:53.264230 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:53.265428 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0929 16:31:53.265695 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1571,"thread_start_us":2078,"threads_started":1}
I20260929 16:31:53.269081 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:53.233732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46583 (local address 127.5.232.129:46913)
0929 16:31:53.242003 (+ 8271us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:53.242052 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:53.244293 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:53.245503 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:53.245516 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:53.246198 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:53.247092 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:53.247120 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:53.248716 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:53.248723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:53.250842 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:53.250858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:53.252142 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:53.252149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:53.252438 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:53.254195 (+ 1757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:53.254226 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:53.257274 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:53.261896 (+ 4622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:53.261909 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:53.261932 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:53.262600 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:53.263154 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:53.263162 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:53.263169 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:53.263451 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:53.264340 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:53.264352 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:53.264864 (+ 512us) client_negotiation.cc:769] Sending connection context
0929 16:31:53.266842 (+ 1978us) client_negotiation.cc:240] Negotiation successful
0929 16:31:53.267498 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7917,"thread_start_us":7710,"threads_started":1}
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
I20260929 16:31:53.271265 18590 heartbeater.cc:347] Connected to a master server at 127.5.232.190:46583
I20260929 16:31:53.271694 18590 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
I20260929 16:31:53.272626 18590 heartbeater.cc:511] Master 127.5.232.190:46583 requested a full tablet report, sending...
I20260929 16:31:53.275033 18377 ts_manager.cc:194] Registered new tserver with Master: a9f97542d118472c834ea3189d30efc5 (127.5.232.129:36517)
I20260929 16:31:53.276537 18377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:46913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:31:53.306499 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:48541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46583
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:31:53 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699512, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
W20260929 16:31:53.735886 18596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:53.736416 18596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:53.736622 18596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:53.748646 18596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:53.748881 18596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:53.749112 18596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:31:53.765174 18596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.232.148:48541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:53.767081 18596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:53.769018 18596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:53.789289 18603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:53.793736 18604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:53.797677 18596 server_base.cc:1034] running on GCE node
W20260929 16:31:53.798534 18606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:53.799399 18596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:53.800854 18596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:53.802150 18596 hybrid_clock.cc:648] HybridClock initialized: now 1790699513802037 us; error 101 us; skew 500 ppm
I20260929 16:31:53.802557 18596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:53 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699513, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:53.806842 18596 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:31:53.810917 18596 webserver.cc:460] Webserver started at http://127.5.232.130:36561/ using document root <none> and password file <none>
I20260929 16:31:53.811921 18596 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:53.812072 18596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:53.812457 18596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:31:53 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699513, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:53.842336 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "87470b864c064189b6ff4572c03726b1"
format_stamp: "Formatted at 2026-09-29 16:31:53 on dist-test-slave-dssx"
server_key: "37fce2b0b02415d07bfec0e1d8e93803"
server_key_iv: "fbb4678a621cdb9423929522b0d81142"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:53.843684 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "87470b864c064189b6ff4572c03726b1"
format_stamp: "Formatted at 2026-09-29 16:31:53 on dist-test-slave-dssx"
server_key: "37fce2b0b02415d07bfec0e1d8e93803"
server_key_iv: "fbb4678a621cdb9423929522b0d81142"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:53.851418 18596 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.001s
I20260929 16:31:53.877501 18620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:53.879694 18596 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.006s sys 0.005s
I20260929 16:31:53.879918 18596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "87470b864c064189b6ff4572c03726b1"
format_stamp: "Formatted at 2026-09-29 16:31:53 on dist-test-slave-dssx"
server_key: "37fce2b0b02415d07bfec0e1d8e93803"
server_key_iv: "fbb4678a621cdb9423929522b0d81142"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:53.880200 18596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:53.900205 18596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:53.905552 18596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:53.905920 18596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:53.907128 18596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:53.909204 18596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:53.909309 18596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:53.909503 18596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:53.909596 18596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:53.962913 18596 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45469
I20260929 16:31:53.962939 18732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45469 every 8 connection(s)
I20260929 16:31:53.965077 18596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:53.970983 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 18596
I20260929 16:31:53.971190 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:31:53.971776 6050 external_mini_cluster.cc:1428] Setting key 1dd6c89a9a0e3ffa51d4eacbf2c31229
Sep 29 16:31:53 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699513, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:31:53.997368 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:53.968161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:34035 (local address 127.5.232.190:46583)
0929 16:31:53.971706 (+ 3545us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:53.971714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:53.976806 (+ 5092us) server_negotiation.cc:407] Connection header received
0929 16:31:53.976880 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:53.976885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:53.976969 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:53.977076 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:53.980540 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:53.981705 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:53.984170 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:53.984782 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:53.984915 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:53.989734 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:53.989741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:53.989746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:53.989811 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:53.993321 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:53.994394 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:53.994400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:53.994404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:53.994560 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:53.995109 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:53.995114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:53.995117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:53.995510 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:53.995640 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:53.996691 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0929 16:31:53.996965 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3392,"thread_start_us":3270,"threads_started":1}
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
I20260929 16:31:53.999960 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:53.969045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46583 (local address 127.5.232.130:34035)
0929 16:31:53.973955 (+ 4910us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:53.974005 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:53.976366 (+ 2361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:53.977576 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:53.977589 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:53.978210 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:53.979083 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:53.979109 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:53.981964 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:53.981969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:53.983991 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:53.984006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:53.985256 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:53.985263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:53.985607 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:53.986439 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:53.986470 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:53.989420 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:53.993564 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:53.993574 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:53.993595 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:53.994238 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:53.994709 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:53.994713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:53.994717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:53.994975 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:53.995732 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:53.995739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:53.996228 (+ 489us) client_negotiation.cc:769] Sending connection context
0929 16:31:53.997908 (+ 1680us) client_negotiation.cc:240] Negotiation successful
0929 16:31:53.998507 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4598,"thread_start_us":4357,"threads_started":1}
I20260929 16:31:54.001940 18733 heartbeater.cc:347] Connected to a master server at 127.5.232.190:46583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
I20260929 16:31:54.002262 18733 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:54.003146 18733 heartbeater.cc:511] Master 127.5.232.190:46583 requested a full tablet report, sending...
I20260929 16:31:54.004908 18377 ts_manager.cc:194] Registered new tserver with Master: 87470b864c064189b6ff4572c03726b1 (127.5.232.130:45469)
I20260929 16:31:54.006127 18377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:34035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:31:54.036230 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:48541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46583
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:31:54.280252 18590 heartbeater.cc:503] Master 127.5.232.190:46583 was elected leader, sending a full tablet report...
W20260929 16:31:54.438892 18739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:54.439227 18739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:54.439404 18739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:54.446580 18739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:54.446749 18739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:54.446933 18739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:31:54.456835 18739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.232.148:48541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:54.458274 18739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:54.459641 18739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:54.473352 18747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:54.473333 18745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:54.473721 18744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:54.474167 18739 server_base.cc:1034] running on GCE node
I20260929 16:31:54.474761 18739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:54.475628 18739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:54.476818 18739 hybrid_clock.cc:648] HybridClock initialized: now 1790699514476763 us; error 58 us; skew 500 ppm
I20260929 16:31:54.477119 18739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:54 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699514, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:54.480286 18739 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:31:54.481806 18739 webserver.cc:460] Webserver started at http://127.5.232.131:39735/ using document root <none> and password file <none>
I20260929 16:31:54.482445 18739 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:54.482545 18739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:31:54.482820 18739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:31:54 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699514, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:54.497879 18739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "6ec2c212fdd648b6a7d52651f90632bf"
format_stamp: "Formatted at 2026-09-29 16:31:54 on dist-test-slave-dssx"
server_key: "4a6084294a16b00fe3a1778d5c69bec7"
server_key_iv: "d46efd987813a7b246318a25a6997484"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:54.498883 18739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "6ec2c212fdd648b6a7d52651f90632bf"
format_stamp: "Formatted at 2026-09-29 16:31:54 on dist-test-slave-dssx"
server_key: "4a6084294a16b00fe3a1778d5c69bec7"
server_key_iv: "d46efd987813a7b246318a25a6997484"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:54.504886 18739 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 16:31:54.522939 18754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:54.524814 18739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:31:54.525007 18739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "6ec2c212fdd648b6a7d52651f90632bf"
format_stamp: "Formatted at 2026-09-29 16:31:54 on dist-test-slave-dssx"
server_key: "4a6084294a16b00fe3a1778d5c69bec7"
server_key_iv: "d46efd987813a7b246318a25a6997484"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:54.525286 18739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:54.544840 18739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:54.548965 18739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:54.549242 18739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:54.550136 18739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:54.551473 18739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:54.551577 18739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:54.551731 18739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:54.551810 18739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:54.584210 18739 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:39511
I20260929 16:31:54.584246 18866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:39511 every 8 connection(s)
I20260929 16:31:54.585616 18739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:54.591409 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 18739
I20260929 16:31:54.591593 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:31:54.592020 6050 external_mini_cluster.cc:1428] Setting key 604aae03603c9a25c98b5da7764394ed
Sep 29 16:31:54 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699514, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:31:54.604661 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:54.588169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:36671 (local address 127.5.232.190:46583)
0929 16:31:54.588646 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:54.588650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:54.589699 (+ 1049us) server_negotiation.cc:407] Connection header received
0929 16:31:54.591443 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:54.591449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:54.591551 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:54.591658 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:54.593296 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.594265 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.595866 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.596264 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.596368 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:54.599567 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:54.599571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:54.599575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:54.599607 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:54.602099 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:54.602816 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:54.602819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:54.602821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:54.602935 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:54.603309 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:54.603312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:54.603314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:54.603555 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:54.603664 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:54.604252 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 16:31:54.604421 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":223,"threads_started":1}
I20260929 16:31:54.605660 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:54.588796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46583 (local address 127.5.232.131:36671)
0929 16:31:54.589536 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:54.589577 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:54.591158 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:54.591900 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:54.591908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:54.592373 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:54.593081 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:54.593101 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.594413 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.594417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:54.595732 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:54.595751 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.596446 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.596450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:54.596639 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:54.597328 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:54.597350 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:54.599360 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:54.602247 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:54.602261 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:54.602280 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:54.602709 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:54.603026 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:54.603029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:54.603031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:54.603211 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:54.603661 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:54.603666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:54.603955 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 16:31:54.604179 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 16:31:54.604544 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
I20260929 16:31:54.607124 18867 heartbeater.cc:347] Connected to a master server at 127.5.232.190:46583
I20260929 16:31:54.607467 18867 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:54.608116 18867 heartbeater.cc:511] Master 127.5.232.190:46583 requested a full tablet report, sending...
I20260929 16:31:54.609436 18377 ts_manager.cc:194] Registered new tserver with Master: 6ec2c212fdd648b6a7d52651f90632bf (127.5.232.131:39511)
I20260929 16:31:54.610199 18377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:36671
I20260929 16:31:54.619405 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:31:54.621052 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 18457
Sep 29 16:31:54 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:31:54.637744 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:54.622093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.5.232.129:36517)
0929 16:31:54.622772 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:54.622779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:54.622804 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:31:54.622859 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:54.622865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:54.623064 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:54.623188 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:54.624575 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.625645 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.627153 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.627714 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.627816 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:54.632022 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:54.632033 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:54.632047 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:54.632079 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:54.634850 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:54.635590 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:54.635597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:54.635600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:54.635737 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:54.636119 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:54.636125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:54.636128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:54.636464 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:54.636781 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:54.637095 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:31:54.637371 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":357,"threads_started":1}
I20260929 16:31:54.638983 18585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:54.722708 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 18457
I20260929 16:31:54.738442 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 18596
Sep 29 16:31:54 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:31:54.754689 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:54.739345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.5.232.130:45469)
0929 16:31:54.739836 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:54.739843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:54.739867 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:31:54.739925 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:54.739931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:54.740140 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:54.740309 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:54.741679 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.742656 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.744022 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.744577 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.744717 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:54.749057 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:54.749070 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:54.749084 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:54.749129 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:54.751798 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:54.752491 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:54.752498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:54.752500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:54.752680 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:54.753139 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:54.753143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:54.753145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:54.753669 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:54.753894 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:54.754199 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 16:31:54.754406 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":242,"threads_started":1}
I20260929 16:31:54.755975 18728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:54.837256 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 18596
I20260929 16:31:54.853709 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 18739
Sep 29 16:31:54 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:31:54.870818 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:54.854669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.5.232.131:39511)
0929 16:31:54.855359 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:54.855367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:54.855395 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:31:54.855454 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:54.855461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:54.855679 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:54.855856 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:54.857237 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.858366 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.859652 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:54.860226 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:54.860367 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:54.865067 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:54.865079 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:54.865095 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:54.865136 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:54.867844 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:54.868562 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:54.868569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:54.868572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:54.868698 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:54.869082 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:54.869089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:54.869092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:54.869509 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:54.869836 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:54.870180 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 16:31:54.870456 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":377,"threads_started":1}
I20260929 16:31:54.872085 18862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:54.955278 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 18739
I20260929 16:31:54.970909 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 18345
I20260929 16:31:54.972105 18407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:55.079885 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 18345
I20260929 16:31:55.104329 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46583
--webserver_interface=127.5.232.190
--webserver_port=34423
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:46583
--ranger_config_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:48541/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.232.148:48541/kms with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:31:55.430120 18879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:55.430415 18879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:55.430610 18879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:55.438066 18879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:31:55.438176 18879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:55.438318 18879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:55.438417 18879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:31:55.438508 18879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:31:55.448632 18879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.232.148:48541/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:46583
--ranger_config_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=34423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:55.449940 18879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:55.451329 18879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:55.463336 18887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:55.463275 18885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:55.463421 18884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:55.464375 18879 server_base.cc:1034] running on GCE node
I20260929 16:31:55.465107 18879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:55.466495 18879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:55.467720 18879 hybrid_clock.cc:648] HybridClock initialized: now 1790699515467679 us; error 53 us; skew 500 ppm
I20260929 16:31:55.468052 18879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:55 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699515, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:55.470978 18879 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:31:55.472386 18879 webserver.cc:460] Webserver started at http://127.5.232.190:34423/ using document root <none> and password file <none>
I20260929 16:31:55.473016 18879 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:55.473157 18879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:31:55 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699515, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:55.527117 18879 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:31:55.530437 18895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:55.531888 18879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:31:55.532089 18879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "920f11fb5a5a4f088aa9c7166edb17f4"
format_stamp: "Formatted at 2026-09-29 16:31:51 on dist-test-slave-dssx"
server_key: "291961f0c3b0d2ae1394c25dd8d2412d"
server_key_iv: "b4cb43f5da5560d83768b3bd2aec05ce"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:55.533560 18879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:55.555351 18879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:55.559239 18879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:55.559533 18879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:55.578037 18879 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:46583
I20260929 16:31:55.578050 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:46583 every 8 connection(s)
I20260929 16:31:55.579571 18879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:31:55.586732 18947 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:31:55.586714 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 18879
I20260929 16:31:55.587786 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:48541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:36517
--local_ip_for_outbound_sockets=127.5.232.129
--tserver_master_addrs=127.5.232.190:46583
--webserver_port=42429
--webserver_interface=127.5.232.129
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.232.148:48541/kms with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:31:55.592046 18947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Bootstrap starting.
I20260929 16:31:55.611639 18947 log.cc:824] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:55.620370 18947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:31:55.620909 18947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Bootstrap complete.
I20260929 16:31:55.630002 18947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } }
I20260929 16:31:55.630841 18947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920f11fb5a5a4f088aa9c7166edb17f4, State: Initialized, Role: FOLLOWER
I20260929 16:31:55.631562 18947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } }
I20260929 16:31:55.631757 18947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:31:55.631896 18947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:31:55.632091 18947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:31:55.636225 18947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } }
I20260929 16:31:55.636718 18947 leader_election.cc:302] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 920f11fb5a5a4f088aa9c7166edb17f4; no voters:
I20260929 16:31:55.637458 18947 leader_election.cc:288] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:31:55.637543 18951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:31:55.639995 18947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:31:55.641662 18951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [term 2 LEADER]: Becoming Leader. State: Replica: 920f11fb5a5a4f088aa9c7166edb17f4, State: Running, Role: LEADER
I20260929 16:31:55.642413 18951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } }
I20260929 16:31:55.645255 18953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 920f11fb5a5a4f088aa9c7166edb17f4. Latest consensus state: current_term: 2 leader_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } } }
I20260929 16:31:55.645701 18953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:55.646552 18960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:31:55.646322 18952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920f11fb5a5a4f088aa9c7166edb17f4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46583 } } }
I20260929 16:31:55.646668 18952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:31:55.660792 18960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:31:55.664544 18960 catalog_manager.cc:1259] Loaded cluster ID: 84737e06063a45ac990661e4defc576f
I20260929 16:31:55.665107 18960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:31:55.670608 18960 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:31:55.672595 18960 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Loaded TSK: 0
I20260929 16:31:55.674616 18960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:31:55.677985 18947 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 16:31:56.054543 18949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:56.054883 18949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:56.055047 18949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:56.065637 18949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:56.065811 18949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:56.066018 18949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:31:56.081177 18949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.232.148:48541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:36517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=42429
--tserver_master_addrs=127.5.232.190:46583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:56.082681 18949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:56.084460 18949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:56.109783 18985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:56.111092 18986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:56.118755 18988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:56.120106 18949 server_base.cc:1034] running on GCE node
I20260929 16:31:56.120868 18949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:56.122062 18949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:56.123310 18949 hybrid_clock.cc:648] HybridClock initialized: now 1790699516123227 us; error 69 us; skew 500 ppm
I20260929 16:31:56.123704 18949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:56 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:56.127571 18949 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:31:56.140550 18949 webserver.cc:460] Webserver started at http://127.5.232.129:42429/ using document root <none> and password file <none>
I20260929 16:31:56.141775 18949 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:56.141953 18949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:31:56 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:56.183645 18949 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.007s sys 0.000s
I20260929 16:31:56.226284 18996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:56.230892 18949 fs_manager.cc:725] Time spent opening block manager: real 0.045s user 0.009s sys 0.031s
I20260929 16:31:56.231259 18949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "a9f97542d118472c834ea3189d30efc5"
format_stamp: "Formatted at 2026-09-29 16:31:52 on dist-test-slave-dssx"
server_key: "2355bf17b370f30f467ff5ec09806f29"
server_key_iv: "02ae7c8d9e49fed2c5a87640a39c909a"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:56.233028 18949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:56.262075 18949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:56.267953 18949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:56.268356 18949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:56.270574 18949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:56.281136 18949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:56.281313 18949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:56.281484 18949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:56.281600 18949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:56.327189 18949 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:36517
I20260929 16:31:56.327253 19109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:36517 every 8 connection(s)
I20260929 16:31:56.328763 18949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:31:56.336494 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 18949
I20260929 16:31:56.337352 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:48541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:45469
--local_ip_for_outbound_sockets=127.5.232.130
--tserver_master_addrs=127.5.232.190:46583
--webserver_port=36561
--webserver_interface=127.5.232.130
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.232.148:48541/kms with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:31:56 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:31:56.365407 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:56.332535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46583 (local address 127.5.232.129:56401)
0929 16:31:56.333427 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:56.333484 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:56.335535 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:56.338247 (+ 2712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:56.338257 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:56.338843 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:56.339659 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:56.339684 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:56.342706 (+ 3022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:56.342713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:56.344861 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:56.344887 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:56.346696 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:56.346704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:56.347040 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:56.348189 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:56.348226 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:56.353229 (+ 5003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:56.358777 (+ 5548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:56.358790 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:56.358813 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:56.359582 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:56.360271 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:56.360279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:56.360286 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:56.360618 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:56.362018 (+ 1400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:56.362030 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:56.362573 (+ 543us) client_negotiation.cc:769] Sending connection context
0929 16:31:56.362953 (+ 380us) client_negotiation.cc:240] Negotiation successful
0929 16:31:56.363666 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":356,"threads_started":1}
I20260929 16:31:56.368116 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:56.332572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:56401 (local address 127.5.232.190:46583)
0929 16:31:56.336388 (+ 3816us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:56.336405 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:56.336449 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:31:56.336922 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:56.336939 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:56.337491 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:56.339995 (+ 2504us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:56.340448 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:56.342457 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:56.345161 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:56.346399 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:56.346646 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:56.353695 (+ 7049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:56.353710 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:56.353738 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:56.353811 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:56.358510 (+ 4699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:56.359804 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:56.359818 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:56.359831 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:56.360098 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:56.360785 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:56.360793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:56.360799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:56.361655 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:56.361901 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:56.366120 (+ 4219us) server_negotiation.cc:299] Negotiation successful
0929 16:31:56.366648 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3551,"thread_start_us":3735,"threads_started":1}
I20260929 16:31:56.372174 19110 heartbeater.cc:347] Connected to a master server at 127.5.232.190:46583
I20260929 16:31:56.372574 19110 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:56.373620 19110 heartbeater.cc:511] Master 127.5.232.190:46583 requested a full tablet report, sending...
W20260929 16:31:56.377359 19110 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:46583 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:31:56.377687 19110 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:56.378002 19110 heartbeater.cc:511] Master 127.5.232.190:46583 requested a full tablet report, sending...
I20260929 16:31:56.379583 19110 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:56.379992 19110 heartbeater.cc:511] Master 127.5.232.190:46583 requested a full tablet report, sending...
W20260929 16:31:56.381249 19110 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:31:56.764238 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:56.764609 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:56.764766 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:56.772749 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:56.772928 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:56.773119 19114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:31:56.784420 19114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.232.148:48541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:45469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=36561
--tserver_master_addrs=127.5.232.190:46583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:56.785914 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:56.787295 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:56.802443 19127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:56.802479 19126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:56.803387 19129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:56.804600 19114 server_base.cc:1034] running on GCE node
I20260929 16:31:56.805292 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:56.806377 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:56.807650 19114 hybrid_clock.cc:648] HybridClock initialized: now 1790699516807577 us; error 59 us; skew 500 ppm
I20260929 16:31:56.808117 19114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:56 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:56.816548 19114 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:31:56.818682 19114 webserver.cc:460] Webserver started at http://127.5.232.130:36561/ using document root <none> and password file <none>
I20260929 16:31:56.819518 19114 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:56.819705 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:31:56 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:56.845337 19114 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:31:56.851884 19139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:56.854027 19114 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.007s sys 0.000s
I20260929 16:31:56.854290 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "87470b864c064189b6ff4572c03726b1"
format_stamp: "Formatted at 2026-09-29 16:31:53 on dist-test-slave-dssx"
server_key: "37fce2b0b02415d07bfec0e1d8e93803"
server_key_iv: "fbb4678a621cdb9423929522b0d81142"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:56.855829 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:56.890662 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:56.896040 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:56.896415 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:56.906255 19114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:56.908181 19114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:56.908371 19114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:56.908535 19114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:56.908660 19114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:56.986445 19114 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45469
I20260929 16:31:56.986532 19253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45469 every 8 connection(s)
I20260929 16:31:56.988404 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:31:56.993700 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 19114
I20260929 16:31:56.994477 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.232.148:48541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:39511
--local_ip_for_outbound_sockets=127.5.232.131
--tserver_master_addrs=127.5.232.190:46583
--webserver_port=39735
--webserver_interface=127.5.232.131
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.232.148:48541/kms with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:31:56 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:31:57 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:31:57.053474 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:56.993659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:38997 (local address 127.5.232.190:46583)
0929 16:31:57.005913 (+ 12254us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:57.005921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:57.009625 (+ 3704us) server_negotiation.cc:407] Connection header received
0929 16:31:57.020684 (+ 11059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:57.020692 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:57.020809 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:57.020893 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:57.023970 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.025199 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.033292 (+ 8093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.034018 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.034136 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.045944 (+ 11808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:57.045952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:57.045957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:57.046004 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:57.049672 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:57.050652 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:57.050658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:57.050662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:57.050820 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:57.051355 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:57.051360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:57.051364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:57.051720 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:57.051867 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:57.052831 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 16:31:57.053062 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12098,"thread_start_us":11971,"threads_started":1}
I20260929 16:31:57.055864 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:56.992909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46583 (local address 127.5.232.130:38997)
0929 16:31:57.008298 (+ 15389us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:57.008358 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:57.020241 (+ 11883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:57.021550 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:57.021565 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:57.022267 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:57.023174 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.023204 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.029128 (+ 5924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.029134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.031368 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.031385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.034521 (+ 3136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.034528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.034877 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.042649 (+ 7772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:57.042694 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:57.045614 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:57.049900 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:57.049910 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:57.049926 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:57.050503 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:57.050969 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:57.050973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:57.050977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:57.051224 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:57.051974 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:57.051981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:57.052411 (+ 430us) client_negotiation.cc:769] Sending connection context
0929 16:31:57.054062 (+ 1651us) client_negotiation.cc:240] Negotiation successful
0929 16:31:57.054590 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14649,"thread_start_us":21860,"threads_started":1}
I20260929 16:31:57.060261 19254 heartbeater.cc:347] Connected to a master server at 127.5.232.190:46583
I20260929 16:31:57.060582 19254 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:57.061501 19254 heartbeater.cc:511] Master 127.5.232.190:46583 requested a full tablet report, sending...
I20260929 16:31:57.063946 18912 ts_manager.cc:194] Registered new tserver with Master: 87470b864c064189b6ff4572c03726b1 (127.5.232.130:45469)
I20260929 16:31:57.066179 18912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:38997
I20260929 16:31:57.381443 19110 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:57.381925 19110 heartbeater.cc:511] Master 127.5.232.190:46583 requested a full tablet report, sending...
I20260929 16:31:57.383359 18912 ts_manager.cc:194] Registered new tserver with Master: a9f97542d118472c834ea3189d30efc5 (127.5.232.129:36517)
I20260929 16:31:57.384518 18912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:56401
W20260929 16:31:57.410213 19258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:31:57.410650 19258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:31:57.410835 19258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:31:57.418557 19258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:31:57.418700 19258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:31:57.418890 19258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:31:57.430826 19258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.232.148:48541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:39511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=39735
--tserver_master_addrs=127.5.232.190:46583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:31:57.432153 19258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:31:57.433727 19258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:31:57.451902 19264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:31:57.452735 19265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 29 16:31:57 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
W20260929 16:31:57.462869 19267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:31:57.467029 19258 server_base.cc:1034] running on GCE node
I20260929 16:31:57.468037 19258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:31:57.471158 19258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:31:57.477736 19258 hybrid_clock.cc:648] HybridClock initialized: now 1790699517477644 us; error 62 us; skew 500 ppm
I20260929 16:31:57.478236 19258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:31:57 dist-test-slave-dssx krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:31:57.482656 19258 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:31:57.549870 19258 webserver.cc:460] Webserver started at http://127.5.232.131:39735/ using document root <none> and password file <none>
I20260929 16:31:57.551046 19258 fs_manager.cc:359] Metadata directory not provided
I20260929 16:31:57.551259 19258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:31:57 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:57.599625 19258 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.008s sys 0.008s
I20260929 16:31:57.609609 19275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:57.611930 19258 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.001s
I20260929 16:31:57.612243 19258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "6ec2c212fdd648b6a7d52651f90632bf"
format_stamp: "Formatted at 2026-09-29 16:31:54 on dist-test-slave-dssx"
server_key: "4a6084294a16b00fe3a1778d5c69bec7"
server_key_iv: "d46efd987813a7b246318a25a6997484"
server_key_version: "kuduclusterkey@0"
I20260929 16:31:57.614058 19258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:31:57.661307 19258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:31:57.666739 19258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:31:57.667089 19258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:31:57.668385 19258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:31:57.670194 19258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:31:57.670311 19258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:57.670420 19258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:31:57.670506 19258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:31:57.711103 19258 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:39511
I20260929 16:31:57.711185 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:39511 every 8 connection(s)
I20260929 16:31:57.712961 19258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:31:57.715389 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 19258
I20260929 16:31:57.759053 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.716435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.5.232.190:46583)
0929 16:31:57.734400 (+ 17965us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:57.734410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:57.736864 (+ 2454us) server_negotiation.cc:407] Connection header received
0929 16:31:57.736991 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:57.736999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:57.737132 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:57.737261 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:57.742478 (+ 5217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.744913 (+ 2435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.748057 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.748726 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.749034 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.752805 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:57.752809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:57.752813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:57.752847 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:57.756010 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:57.756889 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:57.756895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:57.756898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:57.757046 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:57.757480 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:57.757485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:57.757488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:57.757835 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:57.758101 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:57.758419 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 16:31:57.758649 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":17766,"thread_start_us":17620,"threads_started":1}
Sep 29 16:31:57 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:31:57.768364 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.729783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:38591 (local address 127.5.232.190:46583)
0929 16:31:57.745804 (+ 16021us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:57.745809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:57.745825 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:31:57.746054 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:57.746059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:57.746153 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:57.746266 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:57.748839 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.750135 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.752593 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.753337 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.753457 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.761418 (+ 7961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:57.761425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:57.761429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:57.761463 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:57.764782 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:57.765721 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:57.765726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:57.765729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:57.765872 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:57.766382 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:57.766386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:57.766389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:57.766722 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:57.766854 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:57.767830 (+ 976us) server_negotiation.cc:299] Negotiation successful
0929 16:31:57.768031 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15900,"thread_start_us":15746,"threads_started":1}
I20260929 16:31:57.770581 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.729859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46583 (local address 127.5.232.131:38591)
0929 16:31:57.743079 (+ 13220us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:57.743140 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:57.745640 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:57.746818 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:57.746832 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:57.747657 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:31:57.748555 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.748582 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.750350 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.750355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.752419 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.752434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.753841 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.753850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.754138 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.756283 (+ 2145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:31:57.756321 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:31:57.761088 (+ 4767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:31:57.764944 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:57.764954 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:57.764974 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:57.765603 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:57.765991 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:31:57.765996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:31:57.766000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:31:57.766256 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:31:57.766936 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:31:57.766944 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:31:57.767397 (+ 453us) client_negotiation.cc:769] Sending connection context
0929 16:31:57.768658 (+ 1261us) client_negotiation.cc:240] Negotiation successful
0929 16:31:57.769214 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12827,"thread_start_us":18045,"threads_started":1}
I20260929 16:31:57.772248 19388 heartbeater.cc:347] Connected to a master server at 127.5.232.190:46583
I20260929 16:31:57.772589 19388 heartbeater.cc:464] Registering TS with master...
I20260929 16:31:57.773519 19388 heartbeater.cc:511] Master 127.5.232.190:46583 requested a full tablet report, sending...
I20260929 16:31:57.775408 18912 ts_manager.cc:194] Registered new tserver with Master: 6ec2c212fdd648b6a7d52651f90632bf (127.5.232.131:39511)
I20260929 16:31:57.776510 18912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:38591
I20260929 16:31:57.777859 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:31:57.798518 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.782012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.5.232.190:46583)
0929 16:31:57.782183 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:57.782190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:57.782343 (+ 153us) server_negotiation.cc:407] Connection header received
0929 16:31:57.782604 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:57.782610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:57.782684 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:57.782802 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:57.784492 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.785612 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.787626 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.788245 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.788343 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.792005 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:57.792012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:57.792017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:57.792055 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:57.795578 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:57.796398 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:57.796403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:57.796407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:57.796546 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:57.797021 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:57.797027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:57.797030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:57.797397 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:57.797596 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:57.798030 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 16:31:57.798226 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:31:57.803671 18912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:31:57.814080 18912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 29 16:31:57 dist-test-slave-dssx krb5kdc[17901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for HTTP/127.5.232.148@KRBTEST.COM
I20260929 16:31:57.962463 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.891069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45469 (local address 127.0.0.1:36842)
0929 16:31:57.939739 (+ 48670us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:57.939801 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:57.940111 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:57.941436 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:57.941448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:57.941610 (+ 162us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:57.942123 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.942140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.944845 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.944854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.948130 (+ 3276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.948157 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.960151 (+ 11994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.960166 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.960529 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.960618 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 16:31:57.960868 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 16:31:57.961209 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48103,"thread_start_us":44671,"threads_started":1}
I20260929 16:31:57.962456 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.881240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.5.232.129:36517)
0929 16:31:57.901421 (+ 20181us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:57.901431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:57.942285 (+ 40854us) server_negotiation.cc:407] Connection header received
0929 16:31:57.942408 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:57.942416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:57.942654 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:57.942870 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:57.946175 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.947735 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.958896 (+ 11161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.960632 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.960797 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.961059 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:57.961885 (+ 826us) server_negotiation.cc:299] Negotiation successful
0929 16:31:57.962065 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19952,"thread_start_us":20147,"threads_started":1}
I20260929 16:31:57.963171 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.891736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.5.232.130:45469)
0929 16:31:57.935473 (+ 43737us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:57.935487 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:57.939980 (+ 4493us) server_negotiation.cc:407] Connection header received
0929 16:31:57.940346 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:57.940359 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:57.940899 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:57.941361 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:57.942529 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.944604 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.949924 (+ 5320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.951977 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.952151 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.952433 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:57.961560 (+ 9127us) server_negotiation.cc:299] Negotiation successful
0929 16:31:57.961858 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43499,"thread_start_us":43746,"threads_started":1}
I20260929 16:31:57.963310 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.880778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36517 (local address 127.0.0.1:44888)
0929 16:31:57.938538 (+ 57760us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:57.938568 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:57.938779 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:57.945473 (+ 6694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:57.945479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:57.945510 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:57.945931 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.945943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.947939 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.947943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.951331 (+ 3388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.951346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.961455 (+ 10109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.961465 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.961649 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.961682 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:31:57.962796 (+ 1114us) client_negotiation.cc:240] Negotiation successful
0929 16:31:57.962964 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57589,"mutex_wait_us":397,"thread_start_us":58142,"threads_started":1}
I20260929 16:31:57.967440 19189 tablet_service.cc:1467] Processing CreateTablet for tablet adc0ce6839934f6d9966d5d8c0ce15f3 (DEFAULT_TABLE table=test-table [id=3e32f6cf93234657929eb185133b3838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:57.967440 19045 tablet_service.cc:1467] Processing CreateTablet for tablet adc0ce6839934f6d9966d5d8c0ce15f3 (DEFAULT_TABLE table=test-table [id=3e32f6cf93234657929eb185133b3838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:57.969326 19045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc0ce6839934f6d9966d5d8c0ce15f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:57.969310 19189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc0ce6839934f6d9966d5d8c0ce15f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:57.995592 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.880749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:39511 (local address 127.0.0.1:39220)
0929 16:31:57.934301 (+ 53552us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:57.934383 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:57.934676 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:57.982058 (+ 47382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:57.982064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:57.982097 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:57.982485 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.982496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.990567 (+ 8071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.990574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.992723 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:57.992741 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.994762 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.994770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:57.994966 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.995002 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:31:57.995079 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:31:57.995244 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53094,"thread_start_us":67526,"threads_started":1}
I20260929 16:31:57.997118 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:57.881491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.5.232.131:39511)
0929 16:31:57.962545 (+ 81054us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:57.962555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:57.964285 (+ 1730us) server_negotiation.cc:407] Connection header received
0929 16:31:57.964413 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:57.964424 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:57.981676 (+ 17252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:57.982703 (+ 1027us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:57.983013 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.984673 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.992966 (+ 8293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:57.994510 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:57.996237 (+ 1727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:57.996499 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:57.996577 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 16:31:57.996759 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80863,"thread_start_us":80692,"threads_started":1}
I20260929 16:31:58.001583 19323 tablet_service.cc:1467] Processing CreateTablet for tablet adc0ce6839934f6d9966d5d8c0ce15f3 (DEFAULT_TABLE table=test-table [id=3e32f6cf93234657929eb185133b3838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:31:58.003185 19323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adc0ce6839934f6d9966d5d8c0ce15f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:31:58.019433 19410 tablet_bootstrap.cc:492] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: Bootstrap starting.
I20260929 16:31:58.033967 19411 tablet_bootstrap.cc:492] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: Bootstrap starting.
I20260929 16:31:58.035095 19410 tablet_bootstrap.cc:654] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:58.036520 19410 log.cc:824] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:58.038056 19411 tablet_bootstrap.cc:654] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:58.038872 19412 tablet_bootstrap.cc:492] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: Bootstrap starting.
I20260929 16:31:58.040143 19411 log.cc:824] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:58.043267 19412 tablet_bootstrap.cc:654] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: Neither blocks nor log segments found. Creating new log.
I20260929 16:31:58.043712 19410 tablet_bootstrap.cc:492] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: No bootstrap required, opened a new log
I20260929 16:31:58.044037 19410 ts_tablet_manager.cc:1397] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.005s
I20260929 16:31:58.044505 19412 log.cc:824] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: Log is configured to *not* fsync() on all Append() calls
I20260929 16:31:58.063902 19411 tablet_bootstrap.cc:492] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: No bootstrap required, opened a new log
I20260929 16:31:58.064477 19411 ts_tablet_manager.cc:1397] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.000s
I20260929 16:31:58.070762 19254 heartbeater.cc:503] Master 127.5.232.190:46583 was elected leader, sending a full tablet report...
I20260929 16:31:58.071461 19410 raft_consensus.cc:348] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } }
I20260929 16:31:58.071951 19410 raft_consensus.cc:374] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:58.072049 19410 raft_consensus.cc:729] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87470b864c064189b6ff4572c03726b1, State: Initialized, Role: FOLLOWER
I20260929 16:31:58.075524 19412 tablet_bootstrap.cc:492] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: No bootstrap required, opened a new log
I20260929 16:31:58.076195 19410 consensus_queue.cc:260] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } }
I20260929 16:31:58.076960 19412 ts_tablet_manager.cc:1397] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.027s
I20260929 16:31:58.084553 19410 ts_tablet_manager.cc:1428] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: Time spent starting tablet: real 0.040s user 0.013s sys 0.009s
I20260929 16:31:58.087646 19411 raft_consensus.cc:348] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } }
I20260929 16:31:58.088158 19411 raft_consensus.cc:374] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:58.088267 19411 raft_consensus.cc:729] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f97542d118472c834ea3189d30efc5, State: Initialized, Role: FOLLOWER
I20260929 16:31:58.089052 19411 consensus_queue.cc:260] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } }
I20260929 16:31:58.090248 19110 heartbeater.cc:503] Master 127.5.232.190:46583 was elected leader, sending a full tablet report...
I20260929 16:31:58.090988 19411 ts_tablet_manager.cc:1428] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: Time spent starting tablet: real 0.026s user 0.008s sys 0.007s
I20260929 16:31:58.090744 19412 raft_consensus.cc:348] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } }
I20260929 16:31:58.091194 19412 raft_consensus.cc:374] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:31:58.091281 19412 raft_consensus.cc:729] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec2c212fdd648b6a7d52651f90632bf, State: Initialized, Role: FOLLOWER
I20260929 16:31:58.092020 19412 consensus_queue.cc:260] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } }
I20260929 16:31:58.093268 19388 heartbeater.cc:503] Master 127.5.232.190:46583 was elected leader, sending a full tablet report...
I20260929 16:31:58.093832 19412 ts_tablet_manager.cc:1428] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260929 16:31:58.120200 19425 raft_consensus.cc:482] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:31:58.120533 19425 raft_consensus.cc:504] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } }
I20260929 16:31:58.122407 19425 leader_election.cc:288] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9f97542d118472c834ea3189d30efc5 (127.5.232.129:36517), 87470b864c064189b6ff4572c03726b1 (127.5.232.130:45469)
I20260929 16:31:58.130070 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:58.122984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:50939 (local address 127.5.232.129:36517)
0929 16:31:58.123226 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:58.123232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:58.123253 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:31:58.123338 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:58.123342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:58.123406 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:58.123483 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:58.124013 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.125050 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.127656 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.128922 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.129114 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:58.129199 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:58.129659 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 16:31:58.129800 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:31:58.130074 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:58.123054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:49471 (local address 127.5.232.130:45469)
0929 16:31:58.123265 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:58.123271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:58.123654 (+ 383us) server_negotiation.cc:407] Connection header received
0929 16:31:58.123737 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:58.123740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:58.123799 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:58.123876 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:31:58.124421 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.125365 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.127817 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.129058 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.129195 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:58.129269 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:58.129659 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 16:31:58.129800 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:31:58.130151 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:58.122619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36517 (local address 127.5.232.131:50939)
0929 16:31:58.122799 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:58.122823 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:58.122959 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:58.123555 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:58.123562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:58.123598 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:58.123823 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:58.123833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.125219 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.125223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:58.127454 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:58.127473 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.129153 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.129162 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:58.129413 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:58.129498 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 16:31:58.129611 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 16:31:58.129817 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":71}
I20260929 16:31:58.130273 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:58.122677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45469 (local address 127.5.232.131:49471)
0929 16:31:58.123502 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 16:31:58.123522 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:31:58.123630 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:31:58.123918 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:31:58.123924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:31:58.123952 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:31:58.124257 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:58.124266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.125561 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.125566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:58.127603 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:31:58.127615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.129267 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.129272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:31:58.129508 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:58.129534 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:31:58.129613 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:31:58.129817 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":44,"thread_start_us":481,"threads_started":1}
I20260929 16:31:58.131471 19209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc0ce6839934f6d9966d5d8c0ce15f3" candidate_uuid: "6ec2c212fdd648b6a7d52651f90632bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87470b864c064189b6ff4572c03726b1" is_pre_election: true
I20260929 16:31:58.131471 19065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc0ce6839934f6d9966d5d8c0ce15f3" candidate_uuid: "6ec2c212fdd648b6a7d52651f90632bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f97542d118472c834ea3189d30efc5" is_pre_election: true
I20260929 16:31:58.132099 19065 raft_consensus.cc:2454] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ec2c212fdd648b6a7d52651f90632bf in term 0.
I20260929 16:31:58.132099 19209 raft_consensus.cc:2454] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ec2c212fdd648b6a7d52651f90632bf in term 0.
I20260929 16:31:58.132877 19279 leader_election.cc:302] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ec2c212fdd648b6a7d52651f90632bf, a9f97542d118472c834ea3189d30efc5; no voters:
I20260929 16:31:58.133306 19425 raft_consensus.cc:2781] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:31:58.133508 19425 raft_consensus.cc:482] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:31:58.133625 19425 raft_consensus.cc:3037] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:58.135746 19425 raft_consensus.cc:504] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } }
I20260929 16:31:58.136379 19425 leader_election.cc:288] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [CANDIDATE]: Term 1 election: Requested vote from peers a9f97542d118472c834ea3189d30efc5 (127.5.232.129:36517), 87470b864c064189b6ff4572c03726b1 (127.5.232.130:45469)
I20260929 16:31:58.136866 19065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc0ce6839934f6d9966d5d8c0ce15f3" candidate_uuid: "6ec2c212fdd648b6a7d52651f90632bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f97542d118472c834ea3189d30efc5"
I20260929 16:31:58.136933 19209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adc0ce6839934f6d9966d5d8c0ce15f3" candidate_uuid: "6ec2c212fdd648b6a7d52651f90632bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87470b864c064189b6ff4572c03726b1"
I20260929 16:31:58.137101 19065 raft_consensus.cc:3037] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:58.137123 19209 raft_consensus.cc:3037] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:31:58.140461 19209 raft_consensus.cc:2454] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ec2c212fdd648b6a7d52651f90632bf in term 1.
I20260929 16:31:58.140461 19065 raft_consensus.cc:2454] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ec2c212fdd648b6a7d52651f90632bf in term 1.
I20260929 16:31:58.140991 19279 leader_election.cc:302] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ec2c212fdd648b6a7d52651f90632bf, a9f97542d118472c834ea3189d30efc5; no voters:
I20260929 16:31:58.141314 19425 raft_consensus.cc:2781] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:31:58.141937 19425 raft_consensus.cc:686] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 1 LEADER]: Becoming Leader. State: Replica: 6ec2c212fdd648b6a7d52651f90632bf, State: Running, Role: LEADER
I20260929 16:31:58.142452 19425 consensus_queue.cc:237] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } }
I20260929 16:31:58.147403 18911 catalog_manager.cc:5495] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ec2c212fdd648b6a7d52651f90632bf (127.5.232.131). New cstate: current_term: 1 leader_uuid: "6ec2c212fdd648b6a7d52651f90632bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ec2c212fdd648b6a7d52651f90632bf" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 39511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 } health_report { overall_health: UNKNOWN } } }
I20260929 16:31:58.201136 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:58.195681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.5.232.131:39511)
0929 16:31:58.196019 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:58.196025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:58.196046 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:31:58.196267 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:58.196271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:58.196359 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:58.196528 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:31:58.196913 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.197720 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.199159 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.199549 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.199649 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:58.200138 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:31:58.200658 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:31:58.200777 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:58.200861 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 16:31:58.200936 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260929 16:31:58.209645 19065 raft_consensus.cc:1261] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [term 1 FOLLOWER]: Refusing update from remote peer 6ec2c212fdd648b6a7d52651f90632bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:58.209645 19209 raft_consensus.cc:1261] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [term 1 FOLLOWER]: Refusing update from remote peer 6ec2c212fdd648b6a7d52651f90632bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:31:58.210711 19427 consensus_queue.cc:1035] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f97542d118472c834ea3189d30efc5" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:58.211550 19425 consensus_queue.cc:1035] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "87470b864c064189b6ff4572c03726b1" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:31:58.223609 19430 mvcc.cc:205] Tried to move back new op lower bound from 7334705226575011840 to 7334705226314887168. Current Snapshot: MvccSnapshot[applied={T|T < 7334705226575011840}]
I20260929 16:31:58.240751 18912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:58.241122 18912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:31:58.245155 18912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 920f11fb5a5a4f088aa9c7166edb17f4: Sending DeleteTablet for 3 replicas of tablet adc0ce6839934f6d9966d5d8c0ce15f3
I20260929 16:31:58.246454 19045 tablet_service.cc:1514] Processing DeleteTablet for tablet adc0ce6839934f6d9966d5d8c0ce15f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:58 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:44888
I20260929 16:31:58.246593 19323 tablet_service.cc:1514] Processing DeleteTablet for tablet adc0ce6839934f6d9966d5d8c0ce15f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:58 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:39220
I20260929 16:31:58.247324 19189 tablet_service.cc:1514] Processing DeleteTablet for tablet adc0ce6839934f6d9966d5d8c0ce15f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:31:58 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:36842
I20260929 16:31:58.247498 19440 tablet_replica.cc:330] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: stopping tablet replica
I20260929 16:31:58.248082 19440 raft_consensus.cc:2229] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:31:58.248922 19440 raft_consensus.cc:2258] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:58.249398 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 18949
I20260929 16:31:58.249948 19439 tablet_replica.cc:330] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: stopping tablet replica
I20260929 16:31:58.249957 19441 tablet_replica.cc:330] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: stopping tablet replica
I20260929 16:31:58.250403 19439 raft_consensus.cc:2229] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:58.250370 19441 raft_consensus.cc:2229] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:31:58.251065 19439 raft_consensus.cc:2258] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:58.251132 19441 raft_consensus.cc:2258] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:31:58.253752 19441 ts_tablet_manager.cc:1905] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:58.254210 19440 ts_tablet_manager.cc:1905] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:58.261332 19441 ts_tablet_manager.cc:1918] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:58.261600 19441 log.cc:1196] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/adc0ce6839934f6d9966d5d8c0ce15f3
I20260929 16:31:58.262180 19441 ts_tablet_manager.cc:1939] T adc0ce6839934f6d9966d5d8c0ce15f3 P 87470b864c064189b6ff4572c03726b1: Deleting consensus metadata
I20260929 16:31:58.262522 19439 ts_tablet_manager.cc:1905] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:31:58.264169 18897 catalog_manager.cc:4841] TS 87470b864c064189b6ff4572c03726b1 (127.5.232.130:45469): tablet adc0ce6839934f6d9966d5d8c0ce15f3 (table test-table [id=3e32f6cf93234657929eb185133b3838]) successfully deleted
I20260929 16:31:58.268416 19440 ts_tablet_manager.cc:1918] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:58.268633 19440 log.cc:1196] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/adc0ce6839934f6d9966d5d8c0ce15f3
I20260929 16:31:58.269269 19440 ts_tablet_manager.cc:1939] T adc0ce6839934f6d9966d5d8c0ce15f3 P 6ec2c212fdd648b6a7d52651f90632bf: Deleting consensus metadata
I20260929 16:31:58.270249 19439 ts_tablet_manager.cc:1918] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:31:58.270429 19439 log.cc:1196] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/adc0ce6839934f6d9966d5d8c0ce15f3
I20260929 16:31:58.270992 19439 ts_tablet_manager.cc:1939] T adc0ce6839934f6d9966d5d8c0ce15f3 P a9f97542d118472c834ea3189d30efc5: Deleting consensus metadata
I20260929 16:31:58.271243 18898 catalog_manager.cc:4841] TS 6ec2c212fdd648b6a7d52651f90632bf (127.5.232.131:39511): tablet adc0ce6839934f6d9966d5d8c0ce15f3 (table test-table [id=3e32f6cf93234657929eb185133b3838]) successfully deleted
I20260929 16:31:58.272874 18899 catalog_manager.cc:4841] TS a9f97542d118472c834ea3189d30efc5 (127.5.232.129:36517): tablet adc0ce6839934f6d9966d5d8c0ce15f3 (table test-table [id=3e32f6cf93234657929eb185133b3838]) successfully deleted
I20260929 16:31:58.273698 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:58.250615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.5.232.129:36517)
0929 16:31:58.253734 (+ 3119us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:58.253738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:58.253756 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:31:58.253805 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:58.253808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:58.253916 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:58.254015 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:58.256132 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.256887 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.258814 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.259237 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.259400 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:58.263517 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:58.263529 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:58.263543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:58.263624 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:58.268136 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:58.269195 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:58.269208 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:58.269212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:58.269508 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:58.271558 (+ 2050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:58.271563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:58.271567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:58.272075 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:58.272426 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:58.272802 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 16:31:58.273147 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3017}
I20260929 16:31:58.274344 19105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:58.368605 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 18949
I20260929 16:31:58.392633 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 19114
I20260929 16:31:58.406652 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:58.393548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.5.232.130:45469)
0929 16:31:58.393763 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:58.393769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:58.393790 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:31:58.393947 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:58.393954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:58.394035 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:58.394147 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:58.395458 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.396193 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.397545 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.397951 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.398094 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:58.400697 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:58.400711 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:58.400727 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:58.400762 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:58.403743 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:58.404481 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:58.404488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:58.404491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:58.404628 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:58.405007 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:58.405014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:58.405017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:58.405434 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:58.405746 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:58.406051 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 16:31:58.406268 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:31:58.407306 19249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:58.518752 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 19114
I20260929 16:31:58.535651 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 19258
I20260929 16:31:58.553071 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:31:58.536522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.5.232.131:39511)
0929 16:31:58.536759 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:31:58.536771 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:31:58.536790 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:31:58.537034 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:31:58.537038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:31:58.537172 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:31:58.537274 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:31:58.539161 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.539972 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.542141 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:31:58.542843 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:31:58.542962 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:31:58.546439 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:31:58.546454 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:31:58.546474 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:31:58.546516 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:31:58.549995 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:58.550983 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:58.550991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:58.550994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:58.551153 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:31:58.551600 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:31:58.551603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:31:58.551605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:31:58.551952 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:31:58.552192 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:31:58.552547 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:31:58.552717 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:31:58.553850 19383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:58.691969 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 19258
I20260929 16:31:58.711395 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 18879
I20260929 16:31:58.712563 18942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:31:58.826617 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 18879
I20260929 16:31:58.847923 6050 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:31:59.183319 6050 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:31:59.183497 6050 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:31:59.528460 6050 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:31:59.528 UTC [17931] LOG: received smart shutdown request
2026-09-29 16:31:59.531 UTC [17931] LOG: background worker "logical replication launcher" (PID 17940) exited with exit code 1
2026-09-29 16:31:59.531 UTC [17935] LOG: shutting down
2026-09-29 16:31:59.703 UTC [17931] LOG: database system is shut down
2026-09-29T16:31:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53757 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:31:59 dist-test-slave-dssx krb5kdc[19454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:31:59 dist-test-slave-dssx krb5kdc[19454](info): set up 2 sockets
Sep 29 16:31:59 dist-test-slave-dssx krb5kdc[19454](info): commencing operation
krb5kdc: starting...
W20260929 16:32:00.933979 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:32:01 dist-test-slave-dssx krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:32:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:32:01.166630 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42133
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42133,127.5.232.189:39119,127.5.232.188:40123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:01.501034 19467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:01.501340 19467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:01.501502 19467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:01.509191 19467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:01.509306 19467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:01.509353 19467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:01.509479 19467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:01.509531 19467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:01.519582 19467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42133,127.5.232.189:39119,127.5.232.188:40123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:01.520881 19467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:01.522221 19467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:01.533924 19473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:01.534017 19472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:01.533982 19475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:01.534982 19467 server_base.cc:1034] running on GCE node
I20260929 16:32:01.535822 19467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:01.537309 19467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:01.538585 19467 hybrid_clock.cc:648] HybridClock initialized: now 1790699521538541 us; error 34 us; skew 500 ppm
I20260929 16:32:01.538988 19467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:01 dist-test-slave-dssx krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:01.542526 19467 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:32:01.544023 19467 webserver.cc:460] Webserver started at http://127.5.232.190:38927/ using document root <none> and password file <none>
I20260929 16:32:01.544729 19467 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:01.544853 19467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:01.545125 19467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:01.548521 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "7a472156884e4721a9ab9b43946507b4"
format_stamp: "Formatted at 2026-09-29 16:32:01 on dist-test-slave-dssx"
server_key: "a8bc8d257eb174d40d1a8d1e1f8f4907"
server_key_iv: "66d4a7357f58704cf656ae22351d9aff"
server_key_version: "encryptionkey@0"
I20260929 16:32:01.549427 19467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "7a472156884e4721a9ab9b43946507b4"
format_stamp: "Formatted at 2026-09-29 16:32:01 on dist-test-slave-dssx"
server_key: "a8bc8d257eb174d40d1a8d1e1f8f4907"
server_key_iv: "66d4a7357f58704cf656ae22351d9aff"
server_key_version: "encryptionkey@0"
I20260929 16:32:01.554960 19467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:32:01.558509 19482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:01.559919 19467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:32:01.560117 19467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "7a472156884e4721a9ab9b43946507b4"
format_stamp: "Formatted at 2026-09-29 16:32:01 on dist-test-slave-dssx"
server_key: "a8bc8d257eb174d40d1a8d1e1f8f4907"
server_key_iv: "66d4a7357f58704cf656ae22351d9aff"
server_key_version: "encryptionkey@0"
I20260929 16:32:01.560312 19467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:01.579447 19467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:01.584035 19467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:01.584350 19467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:01.603411 19467 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:42133
I20260929 16:32:01.603498 19533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:42133 every 8 connection(s)
I20260929 16:32:01.604796 19467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:32:01.608557 19534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:01.609809 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 19467
I20260929 16:32:01.610039 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:32:01.610675 6050 external_mini_cluster.cc:1428] Setting key 8296a70f549b5efe2730a73435a5632d
I20260929 16:32:01.615212 19534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:32:01 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.189@KRBTEST.COM" created.
I20260929 16:32:01.639249 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.618404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.5.232.190:42133)
0929 16:32:01.620135 (+ 1731us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:01.620146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:01.620206 (+ 60us) server_negotiation.cc:407] Connection header received
0929 16:32:01.621866 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:01.621877 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:01.622197 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:01.622426 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:01.623586 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:01.625559 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:01.626987 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:01.628013 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:01.628238 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:01.631644 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:01.631652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:01.631664 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:01.631697 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:01.634810 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:01.635503 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:01.635517 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:01.635519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:01.635662 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:01.636048 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:01.636055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:01.636058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:01.636579 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:01.636767 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:01.637334 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 16:32:01.637736 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":1483,"spinlock_wait_cycles":19712,"thread_start_us":1278,"threads_started":1}
I20260929 16:32:01.639282 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.618405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42133 (local address 127.0.0.1:41634)
0929 16:32:01.619417 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:01.619478 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:01.621478 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:01.622459 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:01.622465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:01.623024 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:01.623384 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:01.623402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:01.625733 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:01.625737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:01.626837 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:01.626848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:01.628235 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:01.628239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:01.628458 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:01.629143 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:01.629166 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:01.631429 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:01.634955 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:01.634967 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:01.634980 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:01.635364 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:01.635773 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:01.635780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:01.635783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:01.635942 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:01.636718 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:01.636725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:01.636961 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 16:32:01.637218 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 16:32:01.637854 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":204,"thread_start_us":410,"threads_started":1}
I20260929 16:32:01.642642 19534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:32:01.643790 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.643399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33544)
0929 16:32:01.643570 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:01.643678 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:32:01.644120 19485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
I20260929 16:32:01.645730 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.645335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33558)
0929 16:32:01.645544 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:01.645609 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260929 16:32:01.646031 19534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39119: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab.
Entry for principal kudu/127.5.232.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab.
I20260929 16:32:01.661634 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.189:39119
--webserver_interface=127.5.232.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab
--principal=kudu/127.5.232.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42133,127.5.232.189:39119,127.5.232.188:40123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:32:01.695412 19534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } attempt: 1
I20260929 16:32:01.696710 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.696234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33562)
0929 16:32:01.696445 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:01.696566 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 16:32:01.697999 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.697631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33564)
0929 16:32:01.697774 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:01.697847 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:32:01.698298 19534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39119: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
I20260929 16:32:01.766543 19534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } attempt: 2
I20260929 16:32:01.767769 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.767277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33566)
0929 16:32:01.767530 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:01.767627 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:32:01.768906 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.768550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33578)
0929 16:32:01.768688 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:01.768773 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 16:32:01.769205 19534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39119: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
I20260929 16:32:01.860455 19534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } attempt: 3
I20260929 16:32:01.861783 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.861268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33590)
0929 16:32:01.861533 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:01.861645 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 16:32:01.862953 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:01.862585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33596)
0929 16:32:01.862733 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:01.862808 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 16:32:01.863283 19534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39119: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
W20260929 16:32:02.001461 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:02.001781 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:02.001940 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:32:02.006538 19534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } attempt: 4
I20260929 16:32:02.007740 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.007202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33602)
0929 16:32:02.007514 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.007604 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260929 16:32:02.008746 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.008391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33606)
0929 16:32:02.008543 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.008634 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:32:02.008980 19534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39119: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39119: connect: Connection refused (error 111)
W20260929 16:32:02.009689 19541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:02.009805 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:02.009894 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:02.009989 19541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:02.010076 19541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:02.019906 19541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42133,127.5.232.189:39119,127.5.232.188:40123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab
--principal=kudu/127.5.232.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.189:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.232.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:02.021229 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:02.022539 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:02.034734 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:02.034699 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:02.035285 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:02.035576 19541 server_base.cc:1034] running on GCE node
I20260929 16:32:02.036442 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:02.037973 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:02.039201 19541 hybrid_clock.cc:648] HybridClock initialized: now 1790699522039135 us; error 56 us; skew 500 ppm
I20260929 16:32:02.039584 19541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:02.047297 19541 init.cc:377] Logged in from keytab as kudu/127.5.232.189@KRBTEST.COM (short username kudu)
I20260929 16:32:02.048911 19541 webserver.cc:460] Webserver started at http://127.5.232.189:42665/ using document root <none> and password file <none>
I20260929 16:32:02.049762 19541 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:02.049911 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:02.050247 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:02.053684 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/data/instance:
uuid: "60a10da8935049e7ad2150958f3a8cb3"
format_stamp: "Formatted at 2026-09-29 16:32:02 on dist-test-slave-dssx"
server_key: "6b1b046f4ea6d0adc1cdcb9cd6a03e17"
server_key_iv: "cb33c1e65d20b6e4b2e731999828c356"
server_key_version: "encryptionkey@0"
I20260929 16:32:02.054605 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/wal/instance:
uuid: "60a10da8935049e7ad2150958f3a8cb3"
format_stamp: "Formatted at 2026-09-29 16:32:02 on dist-test-slave-dssx"
server_key: "6b1b046f4ea6d0adc1cdcb9cd6a03e17"
server_key_iv: "cb33c1e65d20b6e4b2e731999828c356"
server_key_version: "encryptionkey@0"
I20260929 16:32:02.060312 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:32:02.063928 19556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:02.065529 19541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:32:02.065728 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/wal
uuid: "60a10da8935049e7ad2150958f3a8cb3"
format_stamp: "Formatted at 2026-09-29 16:32:02 on dist-test-slave-dssx"
server_key: "6b1b046f4ea6d0adc1cdcb9cd6a03e17"
server_key_iv: "cb33c1e65d20b6e4b2e731999828c356"
server_key_version: "encryptionkey@0"
I20260929 16:32:02.065954 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:02.087241 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:02.091445 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:02.091729 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:02.110352 19541 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.189:39119
I20260929 16:32:02.110374 19607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.189:39119 every 8 connection(s)
I20260929 16:32:02.111775 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
I20260929 16:32:02.112959 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 19541
I20260929 16:32:02.113168 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-1/wal/instance
I20260929 16:32:02.113778 6050 external_mini_cluster.cc:1428] Setting key 41312e45648cfa87ebe7e1b6fc8a143d
I20260929 16:32:02.116734 19608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:02.125320 19608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.232.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.188@KRBTEST.COM" created.
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:02.148947 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.128164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.5.232.190:42133)
0929 16:32:02.128343 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.128348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.129823 (+ 1475us) server_negotiation.cc:407] Connection header received
0929 16:32:02.131737 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.131743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.131836 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.131960 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.134363 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.135036 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.137416 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.137909 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.138010 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.141863 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.141873 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.141881 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.141930 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.145741 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.146569 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.146579 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.146582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.146706 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.147166 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.147175 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.147177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.147478 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.147645 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.148359 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.148607 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:32:02.150076 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.128800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42133 (local address 127.0.0.1:41650)
0929 16:32:02.129650 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.129693 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:02.131443 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:02.132296 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:02.132310 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:02.133184 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:02.134165 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.134193 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.135216 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.135224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.137217 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.137230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.138110 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.138115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.138316 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.139055 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:02.139091 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:02.141621 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:02.145924 (+ 4303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.145931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.145960 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.146420 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.146838 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.146842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.146845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.147042 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.147610 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:02.147620 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:02.147992 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 16:32:02.148265 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 16:32:02.148814 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":337,"threads_started":1}
I20260929 16:32:02.151785 19608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:32:02.166306 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.152649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33612)
0929 16:32:02.152841 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.152865 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:02.152988 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:02.153789 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:02.153793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:02.154230 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:02.154355 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.154363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.155689 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.155697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.156513 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.156523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.157320 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.157325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.157612 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.158201 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:02.158217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:02.160417 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:02.163667 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.163674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.163679 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.164113 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.164496 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.164500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.164504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.164630 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.165287 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:02.165291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:02.165484 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 16:32:02.165624 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 16:32:02.165900 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":62}
I20260929 16:32:02.166368 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.152733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.5.232.189:39119)
0929 16:32:02.153323 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.153329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.153354 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:32:02.153447 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.153457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.153650 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.153801 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.154508 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.155558 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.156642 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.157179 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.157310 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.160628 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.160639 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.160653 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.160680 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.163546 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.164242 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.164256 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.164258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.164366 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.164725 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.164733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.164739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.165136 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.165353 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.165760 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.166011 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":454,"thread_start_us":296,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab.
Entry for principal kudu/127.5.232.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab.
I20260929 16:32:02.168872 19608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:32:02.169050 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.188:40123
--webserver_interface=127.5.232.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab
--principal=kudu/127.5.232.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42133,127.5.232.189:39119,127.5.232.188:40123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:32:02.170012 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.169542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42102)
0929 16:32:02.169704 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.169841 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 16:32:02.170272 19559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
I20260929 16:32:02.174630 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.174272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42118)
0929 16:32:02.174449 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.174525 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:32:02.174824 19608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40123: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
I20260929 16:32:02.224146 19608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } attempt: 1
I20260929 16:32:02.225436 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.224921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42126)
0929 16:32:02.225116 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.225222 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:32:02.226440 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.226087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42140)
0929 16:32:02.226255 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.226334 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:32:02.226639 19608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40123: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
I20260929 16:32:02.294890 19608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } attempt: 2
I20260929 16:32:02.296046 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.295573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42156)
0929 16:32:02.295786 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.295893 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:32:02.297026 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.296683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42168)
0929 16:32:02.296836 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.296913 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:32:02.297237 19608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40123: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
I20260929 16:32:02.308259 19534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } attempt: 5
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:32:02.322134 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.309219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.5.232.189:39119)
0929 16:32:02.309471 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.309477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.309503 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:32:02.309566 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.309570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.309629 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.309739 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.310556 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.311270 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.312458 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.312905 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.313050 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.316556 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.316561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.316565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.316599 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.319522 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.320176 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.320182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.320186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.320317 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.320683 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.320689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.320692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.320991 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.321148 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.321584 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.321795 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:32:02.322132 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.308965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33616)
0929 16:32:02.309150 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.309178 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:02.309322 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:02.309782 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:02.309788 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:02.310198 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:02.310400 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.310410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.311410 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.311416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.312295 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.312306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.313061 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.313067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.313252 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.313985 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:02.314003 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:02.316351 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:02.319668 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.319674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.319678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.320053 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.320428 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.320433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.320436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.320574 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.321110 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:02.321115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:02.321278 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 16:32:02.321477 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 16:32:02.321772 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 16:32:02.323408 19534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:32:02.324501 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.324054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42176)
0929 16:32:02.324236 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.324330 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:32:02.325522 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.325102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42188)
0929 16:32:02.325240 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.325341 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:32:02.325770 19534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40123: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
I20260929 16:32:02.377038 19534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } attempt: 1
I20260929 16:32:02.378545 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.377876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42196)
0929 16:32:02.378139 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.378283 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 16:32:02.379608 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.379247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42210)
0929 16:32:02.379397 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.379481 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:32:02.379817 19534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40123: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
I20260929 16:32:02.388428 19608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } attempt: 3
I20260929 16:32:02.389665 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.389134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42220)
0929 16:32:02.389343 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.389507 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 16:32:02.390714 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.390301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42222)
0929 16:32:02.390526 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.390610 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260929 16:32:02.390969 19608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40123: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
I20260929 16:32:02.448071 19534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } attempt: 2
I20260929 16:32:02.449514 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.448813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42238)
0929 16:32:02.449070 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.449214 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:32:02.450644 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.450218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42252)
0929 16:32:02.450413 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.450498 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:32:02.450865 19534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40123: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
W20260929 16:32:02.501024 19615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:02.501330 19615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:02.501567 19615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:02.508880 19615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:02.509004 19615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:02.509099 19615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:02.509186 19615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:02.509271 19615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:02.519213 19615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42133,127.5.232.189:39119,127.5.232.188:40123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab
--principal=kudu/127.5.232.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.188:40123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.232.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:02.520618 19615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:02.521991 19615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:32:02.534171 19608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } attempt: 4
W20260929 16:32:02.534670 19621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:02.535104 19620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:02.535516 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.534906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42262)
0929 16:32:02.535254 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.535354 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
W20260929 16:32:02.535545 19623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:02.536458 19615 server_base.cc:1034] running on GCE node
I20260929 16:32:02.536603 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.536223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42272)
0929 16:32:02.536371 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.536456 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:32:02.536796 19608 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40123: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
I20260929 16:32:02.537552 19615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:02.540069 19615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:02.541240 19615 hybrid_clock.cc:648] HybridClock initialized: now 1790699522541187 us; error 50 us; skew 500 ppm
I20260929 16:32:02.541652 19615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:02.544723 19615 init.cc:377] Logged in from keytab as kudu/127.5.232.188@KRBTEST.COM (short username kudu)
I20260929 16:32:02.546335 19615 webserver.cc:460] Webserver started at http://127.5.232.188:35801/ using document root <none> and password file <none>
I20260929 16:32:02.547004 19615 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:02.547102 19615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:02.547371 19615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:02.550410 19615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/data/instance:
uuid: "09cfae5e938448c0b5b6aea4c07464a2"
format_stamp: "Formatted at 2026-09-29 16:32:02 on dist-test-slave-dssx"
server_key: "aa3a09b3098642443c6f696161a55542"
server_key_iv: "a264dfc6d3c442d19685aedc05af0234"
server_key_version: "encryptionkey@0"
I20260929 16:32:02.551182 19615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/wal/instance:
uuid: "09cfae5e938448c0b5b6aea4c07464a2"
format_stamp: "Formatted at 2026-09-29 16:32:02 on dist-test-slave-dssx"
server_key: "aa3a09b3098642443c6f696161a55542"
server_key_iv: "a264dfc6d3c442d19685aedc05af0234"
server_key_version: "encryptionkey@0"
I20260929 16:32:02.556572 19615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:32:02.557111 19534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } attempt: 3
I20260929 16:32:02.558377 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.557759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42286)
0929 16:32:02.558050 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.558175 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 16:32:02.559508 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.559048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42298)
0929 16:32:02.559232 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.559362 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 16:32:02.559703 19534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40123: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40123: connect: Connection refused (error 111)
I20260929 16:32:02.560945 19630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:02.562758 19615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:32:02.562920 19615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/wal
uuid: "09cfae5e938448c0b5b6aea4c07464a2"
format_stamp: "Formatted at 2026-09-29 16:32:02 on dist-test-slave-dssx"
server_key: "aa3a09b3098642443c6f696161a55542"
server_key_iv: "a264dfc6d3c442d19685aedc05af0234"
server_key_version: "encryptionkey@0"
I20260929 16:32:02.563171 19615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:02.576977 19615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:02.581079 19615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:02.581423 19615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:02.600188 19615 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.188:40123
I20260929 16:32:02.600220 19681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.188:40123 every 8 connection(s)
I20260929 16:32:02.601586 19615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
I20260929 16:32:02.602663 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 19615
I20260929 16:32:02.602885 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699314866137-6050-0/minicluster-data/master-2/wal/instance
I20260929 16:32:02.603615 6050 external_mini_cluster.cc:1428] Setting key 8010239923ac686e1645434b4b8f7f68
I20260929 16:32:02.607630 19682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:02.617311 19682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:02.635299 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.620293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.5.232.190:42133)
0929 16:32:02.620533 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.620541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.621293 (+ 752us) server_negotiation.cc:407] Connection header received
0929 16:32:02.621590 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.621595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.621654 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.621760 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.622914 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.623707 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.625272 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.626078 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.626225 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.629787 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.629793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.629796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.629829 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.632510 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.633216 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.633228 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.633230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.633360 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.633780 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.633786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.633789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.634037 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.634196 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.634715 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.634924 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:32:02.635589 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.620322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42133 (local address 127.0.0.1:41652)
0929 16:32:02.621169 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.621218 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:02.621456 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:02.621828 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:02.621834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:02.622402 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:02.622757 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.622768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.623855 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.623859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.625108 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.625125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.626320 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.626328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.626530 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.627192 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:02.627212 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:02.629585 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:02.632644 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.632657 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.632665 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.633085 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.633520 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.633529 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.633533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.633668 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.634162 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:02.634171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:02.634394 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 16:32:02.634615 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 16:32:02.634924 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":421,"threads_started":1}
I20260929 16:32:02.635612 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.607144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.5.232.188:40123)
0929 16:32:02.614088 (+ 6944us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.614110 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.614651 (+ 541us) server_negotiation.cc:407] Connection header received
0929 16:32:02.616926 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.616953 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.617489 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.617957 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.619360 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.621112 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.622577 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.623340 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.623469 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.628414 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.628422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.628433 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.628461 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.631422 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.632170 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.632179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.632188 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.632338 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.632753 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.632757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.632761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.633292 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.633630 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.634085 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.634509 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6668,"thread_start_us":6373,"threads_started":1}
I20260929 16:32:02.637202 19682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:02.651901 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.638035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.5.232.189:39119)
0929 16:32:02.638219 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.638225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.638245 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:32:02.638306 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.638311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.638379 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.638488 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.639242 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.640352 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.641983 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.642610 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.642736 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.646006 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.646015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.646021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.646067 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.649115 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.649970 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.649974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.649976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.650105 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.650499 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.650502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.650504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.650841 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.650976 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.651455 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.651625 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:32:02.651963 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.637925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39119 (local address 127.0.0.1:33618)
0929 16:32:02.638090 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.638112 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:02.638210 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:02.638507 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:02.638511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:02.638904 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:02.639084 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.639095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.640506 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.640510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.641839 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.641852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.642756 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.642761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.642922 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.643478 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:02.643494 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:02.645793 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:02.649242 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.649247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.649251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.649812 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.650218 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.650222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.650226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.650390 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.650970 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:02.650976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:02.651164 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 16:32:02.651313 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 16:32:02.651595 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:32:02.653050 19682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:32:02.655339 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.638964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.5.232.190:42133)
0929 16:32:02.639127 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.639134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.639153 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:32:02.639332 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.639335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.639402 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.639471 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.640881 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.641975 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.643656 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.644296 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.644448 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.649056 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.649061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.649064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.649096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.652361 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.653143 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.653147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.653149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.653321 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.653800 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.653805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.653809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.654183 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.654399 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.654767 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.654993 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:32:02.667726 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.654047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.5.232.188:40123)
0929 16:32:02.654209 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.654213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.654225 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:32:02.654284 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.654289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.654359 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.654452 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.655252 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.656040 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.657451 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.657909 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.658014 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.662487 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.662492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.662495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.662529 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.665172 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.665884 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.665891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.665900 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.666005 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.666350 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.666358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.666361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.666636 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.666786 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.667138 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.667365 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:32:02.667914 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.653815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42312)
0929 16:32:02.653968 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.653983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:02.654053 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:02.654499 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:02.654512 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:02.654966 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:02.655131 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.655141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.656157 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.656165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.657249 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.657270 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.658063 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.658070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.658288 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.659868 (+ 1580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:02.659891 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:02.662293 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:02.665300 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.665307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.665311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.665734 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.666113 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.666121 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.666126 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.666248 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.666748 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:02.666753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:02.666890 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 16:32:02.667044 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:32:02.667545 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":150}
I20260929 16:32:02.672910 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.656915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.5.232.189:39119)
0929 16:32:02.657061 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.657065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.657080 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:32:02.657199 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.657202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.657254 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.657328 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.658818 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.659626 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.661469 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.661969 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.662102 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.666683 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.666689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.666701 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.666734 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.670073 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.670831 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.670843 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.670847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.671010 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.671379 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.671384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.671389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.671734 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.672005 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.672357 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.672591 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 16:32:02.673857 19682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2: Bootstrap starting.
I20260929 16:32:02.676923 19682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:02.678265 19682 log.cc:824] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:02.681349 19682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2: No bootstrap required, opened a new log
I20260929 16:32:02.682333 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:32:02.689836 19682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } }
I20260929 16:32:02.690301 19682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:02.690436 19682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09cfae5e938448c0b5b6aea4c07464a2, State: Initialized, Role: FOLLOWER
I20260929 16:32:02.691254 19682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } }
I20260929 16:32:02.697752 19695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.698146 19695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:32:02.699436 19682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:32:02.708585 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.687662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.5.232.189:39119)
0929 16:32:02.687877 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.687885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.687910 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:32:02.688125 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.688131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.688215 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.688283 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.690826 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.691984 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.695034 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.695968 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.696093 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.701063 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.701072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.701080 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.701139 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.705032 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.705988 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.705994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.705998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.706154 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.706669 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.706675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.706678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.707068 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.707308 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.707957 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.708201 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:32:02.721432 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.687177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.5.232.190:42133)
0929 16:32:02.687377 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.687384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.700182 (+ 12798us) server_negotiation.cc:407] Connection header received
0929 16:32:02.700641 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.700650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.700765 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.700858 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.703356 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.704822 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.707644 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.708629 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.708756 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.713445 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.713454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.713461 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.713504 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.717425 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.718437 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.718442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.718445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.718615 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.719070 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.719074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.719079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.719482 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.720534 (+ 1052us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.720666 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.720995 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:32:02.722352 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.689045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.5.232.188:40123)
0929 16:32:02.689247 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.689255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.702735 (+ 13480us) server_negotiation.cc:407] Connection header received
0929 16:32:02.703253 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.703262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.703359 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.703437 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.705228 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.706356 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.708348 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.709026 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.709131 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.713238 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.713243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.713271 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.713317 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.717809 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.718749 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.718754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.718757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.718921 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.719389 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.719394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.719397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.719786 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.720040 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.721465 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.721933 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 16:32:02.729089 19706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:32:02.729240 19706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:32:02.719830 19615 master_runner.cc:186] Error getting master registration for 127.5.232.190:42133: OK, instance_id { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" instance_seqno: 1790699521597951 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:32:02.733093 19615 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:32:02.736917 19534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } attempt: 4
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
I20260929 16:32:02.752219 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.737503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42328)
0929 16:32:02.737712 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.737734 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:02.737842 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:02.738155 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:02.738160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:02.738694 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:02.738900 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.738912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.739946 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.739951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.741460 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.741476 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.742512 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.742518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.742734 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.743392 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:02.743409 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:02.745467 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:02.749535 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.749541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.749545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.750126 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.750638 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.750643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.750646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.750828 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.751492 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:02.751496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:02.751641 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 16:32:02.751747 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 16:32:02.751952 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:32:02.752522 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.737652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.5.232.188:40123)
0929 16:32:02.737825 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.737829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.737845 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:32:02.737958 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.737961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.738014 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.738089 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.739045 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.739795 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.741643 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.742348 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.742496 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.745657 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.745664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.745670 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.745709 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.749326 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.750308 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.750314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.750317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.750507 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.750962 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.750967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.750971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.751353 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.751518 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.751901 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.752151 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:32:02.758543 19534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4: Bootstrap starting.
I20260929 16:32:02.762753 19534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:02.764122 19534 log.cc:824] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:02.767549 19534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4: No bootstrap required, opened a new log
I20260929 16:32:02.778720 19534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } }
I20260929 16:32:02.779181 19534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:02.779300 19534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a472156884e4721a9ab9b43946507b4, State: Initialized, Role: FOLLOWER
I20260929 16:32:02.779994 19534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } }
I20260929 16:32:02.781298 19709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.781716 19709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:32:02.782662 19534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:32:02.792227 19720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:32:02.792430 19720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:32:02.791553 19467 master_runner.cc:186] Error getting master registration for 127.5.232.189:39119: OK, instance_id { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" instance_seqno: 1790699522104930 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:32:02.803460 19467 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:32:02.805637 19695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:32:02.805959 19695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } }
I20260929 16:32:02.807626 19695 leader_election.cc:288] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a472156884e4721a9ab9b43946507b4 (127.5.232.190:42133), 60a10da8935049e7ad2150958f3a8cb3 (127.5.232.189:39119)
I20260929 16:32:02.809698 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09cfae5e938448c0b5b6aea4c07464a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a472156884e4721a9ab9b43946507b4" is_pre_election: true
I20260929 16:32:02.810286 19509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09cfae5e938448c0b5b6aea4c07464a2 in term 0.
I20260929 16:32:02.810901 19631 leader_election.cc:302] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09cfae5e938448c0b5b6aea4c07464a2, 7a472156884e4721a9ab9b43946507b4; no voters:
I20260929 16:32:02.811358 19695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:32:02.811519 19695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:32:02.811637 19695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:02.815065 19695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } }
I20260929 16:32:02.815785 19695 leader_election.cc:288] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [CANDIDATE]: Term 1 election: Requested vote from peers 7a472156884e4721a9ab9b43946507b4 (127.5.232.190:42133), 60a10da8935049e7ad2150958f3a8cb3 (127.5.232.189:39119)
I20260929 16:32:02.816584 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09cfae5e938448c0b5b6aea4c07464a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a472156884e4721a9ab9b43946507b4"
I20260929 16:32:02.816800 19509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:02.820284 19509 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09cfae5e938448c0b5b6aea4c07464a2 in term 1.
I20260929 16:32:02.820797 19631 leader_election.cc:302] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09cfae5e938448c0b5b6aea4c07464a2, 7a472156884e4721a9ab9b43946507b4; no voters:
I20260929 16:32:02.821113 19695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 1 FOLLOWER]: Leader election won for term 1
W20260929 16:32:02.833712 19636 tablet.cc:2369] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:02.833868 19695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [term 1 LEADER]: Becoming Leader. State: Replica: 09cfae5e938448c0b5b6aea4c07464a2, State: Running, Role: LEADER
I20260929 16:32:02.834550 19695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } }
I20260929 16:32:02.836030 19608 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } attempt: 5
W20260929 16:32:02.837749 19488 tablet.cc:2369] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:32:02 dist-test-slave-dssx krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
I20260929 16:32:02.857627 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.836632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40123 (local address 127.0.0.1:42336)
0929 16:32:02.837031 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:02.837054 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:02.837173 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:02.838201 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:02.838207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:02.838764 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:02.838979 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.838993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.840537 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.840545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.842088 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:02.842106 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.843444 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.843450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:02.843676 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.844390 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:02.844414 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:02.847579 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:02.853061 (+ 5482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.853079 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.853084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.853845 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.854586 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:02.854592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:02.854595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:02.854888 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:02.855989 (+ 1101us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:02.855995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:02.856241 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 16:32:02.856485 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 16:32:02.856941 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260929 16:32:02.860489 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:02.837518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.5.232.188:40123)
0929 16:32:02.837689 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:02.837696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:02.837716 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:32:02.837784 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:02.837790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:02.837878 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:02.837983 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:02.839135 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.840292 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.842320 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:02.843021 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:02.843150 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:02.847888 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:02.847895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:02.847901 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:02.847942 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:02.852813 (+ 4871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.854110 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.854116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.854119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.854383 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:02.855110 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:02.855115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:02.855119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:02.855776 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:02.858487 (+ 2711us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:02.858755 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:32:02.859218 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:32:02.859565 19722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09cfae5e938448c0b5b6aea4c07464a2. Latest consensus state: current_term: 1 leader_uuid: "09cfae5e938448c0b5b6aea4c07464a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.861918 19722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:02.867002 19608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3: Bootstrap starting.
I20260929 16:32:02.872319 19608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:02.874531 19608 log.cc:824] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:02.875896 19724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:32:02.879693 19724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:32:02.884274 19608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3: No bootstrap required, opened a new log
I20260929 16:32:02.895884 19608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } }
I20260929 16:32:02.896342 19608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:02.896471 19608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60a10da8935049e7ad2150958f3a8cb3, State: Initialized, Role: FOLLOWER
I20260929 16:32:02.897264 19608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } }
I20260929 16:32:02.904821 19727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.905153 19509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [term 1 FOLLOWER]: Refusing update from remote peer 09cfae5e938448c0b5b6aea4c07464a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:02.905237 19727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:32:02.907167 19582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09cfae5e938448c0b5b6aea4c07464a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60a10da8935049e7ad2150958f3a8cb3"
I20260929 16:32:02.907223 19583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09cfae5e938448c0b5b6aea4c07464a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60a10da8935049e7ad2150958f3a8cb3" is_pre_election: true
I20260929 16:32:02.907579 19581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:02.907140 19608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:32:02.910905 19722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:02.911405 19581 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [term 1 FOLLOWER]: Refusing update from remote peer 09cfae5e938448c0b5b6aea4c07464a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:02.916115 19722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:02.938920 19727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09cfae5e938448c0b5b6aea4c07464a2. Latest consensus state: current_term: 1 leader_uuid: "09cfae5e938448c0b5b6aea4c07464a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.939277 19727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:32:02.942662 19726 mvcc.cc:205] Tried to move back new op lower bound from 7334705245805260800 to 7334705245535350784. Current Snapshot: MvccSnapshot[applied={T|T < 7334705245805260800}]
I20260929 16:32:02.954036 19727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09cfae5e938448c0b5b6aea4c07464a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.954341 19727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:32:02.960506 19709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09cfae5e938448c0b5b6aea4c07464a2. Latest consensus state: current_term: 1 leader_uuid: "09cfae5e938448c0b5b6aea4c07464a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.960903 19709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:32:02.964533 19695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09cfae5e938448c0b5b6aea4c07464a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.964922 19695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:02.965209 19722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09cfae5e938448c0b5b6aea4c07464a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.965473 19722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:02.966109 19729 mvcc.cc:205] Tried to move back new op lower bound from 7334705245805260800 to 7334705245535350784. Current Snapshot: MvccSnapshot[applied={T|T < 7334705245805260800}]
I20260929 16:32:02.968743 19724 catalog_manager.cc:1347] Generated new cluster ID: 5ce47bfb70c8423e999f9a07896d1963
I20260929 16:32:02.968842 19724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:32:02.969480 19709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09cfae5e938448c0b5b6aea4c07464a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a472156884e4721a9ab9b43946507b4" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42133 } } peers { permanent_uuid: "60a10da8935049e7ad2150958f3a8cb3" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39119 } } peers { permanent_uuid: "09cfae5e938448c0b5b6aea4c07464a2" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40123 } } }
I20260929 16:32:02.969734 19709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:32:02.987862 19724 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:32:02.990679 19724 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:32:03.008806 19724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09cfae5e938448c0b5b6aea4c07464a2: Generated new TSK 0
I20260929 16:32:03.010180 19724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:32:03.032737 19746 catalog_manager.cc:1259] Loaded cluster ID: 5ce47bfb70c8423e999f9a07896d1963
I20260929 16:32:03.032860 19746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3: loading cluster ID for follower catalog manager: success
I20260929 16:32:03.036716 19746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3: acquiring CA information for follower catalog manager: success
I20260929 16:32:03.039564 19746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60a10da8935049e7ad2150958f3a8cb3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:32:03.794454 19720 catalog_manager.cc:1259] Loaded cluster ID: 5ce47bfb70c8423e999f9a07896d1963
I20260929 16:32:03.794610 19720 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4: loading cluster ID for follower catalog manager: success
I20260929 16:32:03.797102 19720 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4: acquiring CA information for follower catalog manager: success
I20260929 16:32:03.799057 19720 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a472156884e4721a9ab9b43946507b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:32:04.185830 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 19467
I20260929 16:32:04.187179 19529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:04.298905 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 19467
I20260929 16:32:04.320607 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 19541
I20260929 16:32:04.321781 19603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:04.429049 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 19541
I20260929 16:32:04.449154 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 19615
I20260929 16:32:04.450284 19677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:04.560564 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 19615
2026-09-29T16:32:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4730 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:32:04 dist-test-slave-dssx krb5kdc[19758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:32:04 dist-test-slave-dssx krb5kdc[19758](info): set up 2 sockets
Sep 29 16:32:04 dist-test-slave-dssx krb5kdc[19758](info): commencing operation
krb5kdc: starting...
W20260929 16:32:05.665990 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:32:05 dist-test-slave-dssx krb5kdc[19758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:32:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:32:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:32:05.908795 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46393
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:06.250160 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:06.250473 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:06.250674 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:06.258097 19771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:06.258235 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:06.258322 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:06.258425 19771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:06.258514 19771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:06.268863 19771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:46393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:06.270260 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:06.271606 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:06.283774 19776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:06.283741 19779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:06.283830 19777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:06.284952 19771 server_base.cc:1034] running on GCE node
I20260929 16:32:06.285907 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:06.287477 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:06.288693 19771 hybrid_clock.cc:648] HybridClock initialized: now 1790699526288643 us; error 38 us; skew 500 ppm
I20260929 16:32:06.289070 19771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:06 dist-test-slave-dssx krb5kdc[19758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699526, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:06.292459 19771 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:32:06.294199 19771 webserver.cc:460] Webserver started at http://127.5.232.190:41523/ using document root <none> and password file <none>
I20260929 16:32:06.294925 19771 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:06.295061 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:06.295404 19771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:06.298532 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "2ad7a0629a2f4be0b982db3c9d8402d6"
format_stamp: "Formatted at 2026-09-29 16:32:06 on dist-test-slave-dssx"
server_key: "a1d01576fc1e5996c330cf4584445bf9"
server_key_iv: "730f44aeadbe6f4726407d90a4475b5e"
server_key_version: "encryptionkey@0"
I20260929 16:32:06.299330 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "2ad7a0629a2f4be0b982db3c9d8402d6"
format_stamp: "Formatted at 2026-09-29 16:32:06 on dist-test-slave-dssx"
server_key: "a1d01576fc1e5996c330cf4584445bf9"
server_key_iv: "730f44aeadbe6f4726407d90a4475b5e"
server_key_version: "encryptionkey@0"
I20260929 16:32:06.304734 19771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:32:06.308557 19786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:06.310071 19771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:32:06.310271 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "2ad7a0629a2f4be0b982db3c9d8402d6"
format_stamp: "Formatted at 2026-09-29 16:32:06 on dist-test-slave-dssx"
server_key: "a1d01576fc1e5996c330cf4584445bf9"
server_key_iv: "730f44aeadbe6f4726407d90a4475b5e"
server_key_version: "encryptionkey@0"
I20260929 16:32:06.310482 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:06.340092 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:06.344493 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:06.344815 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:06.364197 19771 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:46393
I20260929 16:32:06.364198 19837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:46393 every 8 connection(s)
I20260929 16:32:06.365765 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:32:06.369979 19838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:06.372488 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 19771
I20260929 16:32:06.372730 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:32:06.373606 6050 external_mini_cluster.cc:1428] Setting key 8bfa3f5cd63473bce91ae56fae6e71d3
I20260929 16:32:06.382485 19838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6: Bootstrap starting.
I20260929 16:32:06.385826 19838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:06.386869 19838 log.cc:824] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:32:06 dist-test-slave-dssx krb5kdc[19758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:06.390379 19838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6: No bootstrap required, opened a new log
I20260929 16:32:06.397102 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:06.375716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.5.232.190:46393)
0929 16:32:06.376611 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:06.376623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:06.376664 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:32:06.378461 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:06.378487 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:06.378902 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:06.379321 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:06.380868 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:06.382282 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:06.383733 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:06.384449 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:06.384661 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:06.389327 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:06.389335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:06.389351 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:06.389442 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:06.392693 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:06.393667 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:06.393674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:06.393680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:06.393865 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:06.394290 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:06.394294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:06.394296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:06.394855 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:06.395093 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:06.395599 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 16:32:06.396000 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":409,"threads_started":1}
I20260929 16:32:06.399670 19838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad7a0629a2f4be0b982db3c9d8402d6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46393 } }
I20260929 16:32:06.400184 19838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:06.400337 19838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad7a0629a2f4be0b982db3c9d8402d6, State: Initialized, Role: FOLLOWER
I20260929 16:32:06.401166 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:32:06.401069 19838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad7a0629a2f4be0b982db3c9d8402d6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46393 } }
I20260929 16:32:06.401407 19838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:32:06.401542 19838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:32:06.401705 19838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:06.404996 19838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad7a0629a2f4be0b982db3c9d8402d6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46393 } }
I20260929 16:32:06.405678 19838 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ad7a0629a2f4be0b982db3c9d8402d6; no voters:
I20260929 16:32:06.406311 19838 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:32:06.406451 19843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:32:06.407151 19843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [term 1 LEADER]: Becoming Leader. State: Replica: 2ad7a0629a2f4be0b982db3c9d8402d6, State: Running, Role: LEADER
I20260929 16:32:06.407732 19843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad7a0629a2f4be0b982db3c9d8402d6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46393 } }
I20260929 16:32:06.408473 19838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:32:06.410985 19844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ad7a0629a2f4be0b982db3c9d8402d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad7a0629a2f4be0b982db3c9d8402d6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46393 } } }
I20260929 16:32:06.411409 19844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:06.412113 19845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ad7a0629a2f4be0b982db3c9d8402d6. Latest consensus state: current_term: 1 leader_uuid: "2ad7a0629a2f4be0b982db3c9d8402d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad7a0629a2f4be0b982db3c9d8402d6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46393 } } }
I20260929 16:32:06.412366 19845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:06.413031 19852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:32:06.418218 19852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:32:06.426501 19852 catalog_manager.cc:1347] Generated new cluster ID: 7bb3cde13c9e4db0a9eed229352c44e9
I20260929 16:32:06.426638 19852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:32:06.450937 19852 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:32:06.453131 19852 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:32:06.465291 19852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ad7a0629a2f4be0b982db3c9d8402d6: Generated new TSK 0
I20260929 16:32:06.466285 19852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:32:06.514819 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 19771
I20260929 16:32:06.516088 19833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:06.615658 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 19771
2026-09-29T16:32:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2050 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:32:06 dist-test-slave-dssx krb5kdc[19866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:32:06 dist-test-slave-dssx krb5kdc[19866](info): set up 2 sockets
Sep 29 16:32:06 dist-test-slave-dssx krb5kdc[19866](info): commencing operation
krb5kdc: starting...
W20260929 16:32:07.710000 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:32:07 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:32:07.948876 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42935
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:08.278218 19879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:08.278535 19879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:08.278645 19879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:08.285842 19879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:08.285964 19879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:08.286010 19879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:08.286067 19879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:08.286113 19879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:08.296001 19879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:08.297297 19879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:08.298592 19879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:08.310699 19885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:08.310654 19887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:08.310745 19884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:08.311748 19879 server_base.cc:1034] running on GCE node
I20260929 16:32:08.312579 19879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:08.314141 19879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:08.315357 19879 hybrid_clock.cc:648] HybridClock initialized: now 1790699528315294 us; error 59 us; skew 500 ppm
I20260929 16:32:08.315744 19879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:08 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699528, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:08.323458 19879 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:32:08.325194 19879 webserver.cc:460] Webserver started at http://127.5.232.190:35541/ using document root <none> and password file <none>
I20260929 16:32:08.326016 19879 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:08.326175 19879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:08.326488 19879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:08.329741 19879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "e3342ffbcf0d421eb102d6a3cee2d929"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "4c52e4caf8bf994cd9e492e6e4a52da8"
server_key_iv: "8025bebd3de18f9c7e231f7a39547649"
server_key_version: "encryptionkey@0"
I20260929 16:32:08.330595 19879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "e3342ffbcf0d421eb102d6a3cee2d929"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "4c52e4caf8bf994cd9e492e6e4a52da8"
server_key_iv: "8025bebd3de18f9c7e231f7a39547649"
server_key_version: "encryptionkey@0"
I20260929 16:32:08.336134 19879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 16:32:08.339769 19894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:08.341344 19879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:32:08.341688 19879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "e3342ffbcf0d421eb102d6a3cee2d929"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "4c52e4caf8bf994cd9e492e6e4a52da8"
server_key_iv: "8025bebd3de18f9c7e231f7a39547649"
server_key_version: "encryptionkey@0"
I20260929 16:32:08.341923 19879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:08.366674 19879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:08.371327 19879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:08.371675 19879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:08.391613 19879 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:42935
I20260929 16:32:08.391655 19945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:42935 every 8 connection(s)
I20260929 16:32:08.393113 19879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:32:08.397498 19946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:08.401692 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 19879
I20260929 16:32:08.401971 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:32:08.402891 6050 external_mini_cluster.cc:1428] Setting key 6678cee0d295b366f3ceb8ccce8f0782
I20260929 16:32:08.412029 19946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Bootstrap starting.
I20260929 16:32:08.416652 19946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:08.418339 19946 log.cc:824] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:32:08 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:08.421825 19946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: No bootstrap required, opened a new log
I20260929 16:32:08.430612 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:08.405424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.5.232.190:42935)
0929 16:32:08.406296 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:08.406309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:08.406353 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:32:08.408115 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:08.408142 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:08.408610 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:08.409086 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:08.410687 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:08.412048 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:08.414326 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:08.415069 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:08.415221 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:08.420947 (+ 5726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:08.420956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:08.420975 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:08.421019 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:08.425363 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:08.426270 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:08.426280 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:08.426289 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:08.426527 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:08.426953 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:08.426958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:08.426962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:08.427804 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:08.428158 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:08.428576 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 16:32:08.429103 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":398,"threads_started":1}
I20260929 16:32:08.432924 19946 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:08.433465 19946 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:08.433593 19946 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3342ffbcf0d421eb102d6a3cee2d929, State: Initialized, Role: FOLLOWER
I20260929 16:32:08.434273 19946 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:08.434497 19946 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:32:08.434594 19946 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:32:08.434731 19946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:08.436957 19946 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:08.437567 19946 leader_election.cc:302] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3342ffbcf0d421eb102d6a3cee2d929; no voters:
I20260929 16:32:08.438328 19946 leader_election.cc:288] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:32:08.438517 19951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:32:08.439339 19951 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 1 LEADER]: Becoming Leader. State: Replica: e3342ffbcf0d421eb102d6a3cee2d929, State: Running, Role: LEADER
I20260929 16:32:08.440033 19951 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:08.440642 19946 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:32:08.443435 19953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3342ffbcf0d421eb102d6a3cee2d929. Latest consensus state: current_term: 1 leader_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } } }
I20260929 16:32:08.443455 19952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } } }
I20260929 16:32:08.444072 19953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:08.444106 19952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:08.452540 19957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:32:08.458606 19957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:32:08.468978 19957 catalog_manager.cc:1347] Generated new cluster ID: 2bcd1b30b116434a9f0b660fef3f7480
I20260929 16:32:08.469133 19957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:32:08.486290 19957 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:32:08.488306 19957 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:32:08.503376 19957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Generated new TSK 0
I20260929 16:32:08.504612 19957 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:32:08.573226 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42935
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:08.935084 19972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:08.935375 19972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:08.935532 19972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:08.943253 19972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:08.943372 19972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:08.943594 19972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:32:08.953962 19972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:08.955349 19972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:08.956718 19972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:08.970645 19977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:08.970660 19978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:08.970685 19980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:08.971565 19972 server_base.cc:1034] running on GCE node
I20260929 16:32:08.972153 19972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:08.973141 19972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:08.974341 19972 hybrid_clock.cc:648] HybridClock initialized: now 1790699528974272 us; error 59 us; skew 500 ppm
I20260929 16:32:08.974689 19972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:08 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699528, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:08.977981 19972 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:32:08.979490 19972 webserver.cc:460] Webserver started at http://127.5.232.129:41587/ using document root <none> and password file <none>
I20260929 16:32:08.980225 19972 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:08.980329 19972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:08.980683 19972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:08.984988 19972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "944912a5f3834f6db44b26e9ccb334f4"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "fe572d280e2dd3f667986078fbbf8259"
server_key_iv: "d09d10386db0882bf21703e041f4ef2a"
server_key_version: "encryptionkey@0"
I20260929 16:32:08.985874 19972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "944912a5f3834f6db44b26e9ccb334f4"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "fe572d280e2dd3f667986078fbbf8259"
server_key_iv: "d09d10386db0882bf21703e041f4ef2a"
server_key_version: "encryptionkey@0"
I20260929 16:32:08.991384 19972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 16:32:08.995297 19987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:08.996765 19972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:32:08.996970 19972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "944912a5f3834f6db44b26e9ccb334f4"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "fe572d280e2dd3f667986078fbbf8259"
server_key_iv: "d09d10386db0882bf21703e041f4ef2a"
server_key_version: "encryptionkey@0"
I20260929 16:32:08.997206 19972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:09.020704 19972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:09.025246 19972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:09.025610 19972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:09.026620 19972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:09.028007 19972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:09.028112 19972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:09.028265 19972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:09.028337 19972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:09.061668 19972 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:40685
I20260929 16:32:09.061690 20099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:40685 every 8 connection(s)
I20260929 16:32:09.063031 19972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:32:09.070854 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 19972
I20260929 16:32:09.071084 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:32:09.071862 6050 external_mini_cluster.cc:1428] Setting key d47d07022407f9dc4db24a52d195a873
Sep 29 16:32:09 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699528, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:09.085224 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:09.065606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:44869 (local address 127.5.232.190:42935)
0929 16:32:09.066014 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:09.066018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:09.067048 (+ 1030us) server_negotiation.cc:407] Connection header received
0929 16:32:09.068880 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:09.068885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:09.068953 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:09.069053 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:09.070686 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:09.071858 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:09.073722 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:09.074273 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:09.074412 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:09.078209 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:09.078219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:09.078228 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:09.078277 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:09.081836 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:09.082859 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:09.082863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:09.082865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:09.082986 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:09.083479 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:09.083482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:09.083484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:09.083757 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:09.083869 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:09.084788 (+ 919us) server_negotiation.cc:299] Negotiation successful
0929 16:32:09.084963 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":192,"threads_started":1}
I20260929 16:32:09.086860 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:09.066188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:44869)
0929 16:32:09.066886 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:09.066922 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:09.068609 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:09.069272 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:09.069281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:09.069884 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:09.070490 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:09.070508 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:09.072032 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:09.072038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:09.073522 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:09.073539 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:09.074520 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:09.074525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:09.074758 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:09.075458 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:09.075481 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:09.077944 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:09.081991 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:09.082004 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:09.082027 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:09.082726 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:09.083096 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:09.083103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:09.083109 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:09.083366 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:09.083887 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:09.083899 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:09.084370 (+ 471us) client_negotiation.cc:769] Sending connection context
0929 16:32:09.084690 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 16:32:09.085301 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":261,"threads_started":1}
I20260929 16:32:09.088526 20100 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42935
I20260929 16:32:09.088855 20100 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:09.089663 20100 heartbeater.cc:511] Master 127.5.232.190:42935 requested a full tablet report, sending...
I20260929 16:32:09.091864 19911 ts_manager.cc:194] Registered new tserver with Master: 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685)
I20260929 16:32:09.093504 19911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:44869
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:32:09.124940 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42935
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:09.459394 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:09.459898 20106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:09.460072 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:09.467780 20106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:09.467979 20106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:09.468160 20106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:32:09.478367 20106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:09.479807 20106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:09.481207 20106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:09.494840 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:09.494872 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:09.495065 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:09.495568 20106 server_base.cc:1034] running on GCE node
I20260929 16:32:09.496284 20106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:09.497283 20106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:09.498495 20106 hybrid_clock.cc:648] HybridClock initialized: now 1790699529498451 us; error 35 us; skew 500 ppm
I20260929 16:32:09.498809 20106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:09 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699529, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:09.502206 20106 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:32:09.503925 20106 webserver.cc:460] Webserver started at http://127.5.232.130:42403/ using document root <none> and password file <none>
I20260929 16:32:09.504626 20106 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:09.504729 20106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:09.504985 20106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:09.508311 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "75486dabdb8b4ad188350506644cd676"
format_stamp: "Formatted at 2026-09-29 16:32:09 on dist-test-slave-dssx"
server_key: "a0b80b3d61393410547cf9265e86aaa6"
server_key_iv: "6a46d744e67c464f2248e3b489b19a3e"
server_key_version: "encryptionkey@0"
I20260929 16:32:09.509040 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "75486dabdb8b4ad188350506644cd676"
format_stamp: "Formatted at 2026-09-29 16:32:09 on dist-test-slave-dssx"
server_key: "a0b80b3d61393410547cf9265e86aaa6"
server_key_iv: "6a46d744e67c464f2248e3b489b19a3e"
server_key_version: "encryptionkey@0"
I20260929 16:32:09.514386 20106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:32:09.517865 20121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:09.519354 20106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:32:09.519502 20106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "75486dabdb8b4ad188350506644cd676"
format_stamp: "Formatted at 2026-09-29 16:32:09 on dist-test-slave-dssx"
server_key: "a0b80b3d61393410547cf9265e86aaa6"
server_key_iv: "6a46d744e67c464f2248e3b489b19a3e"
server_key_version: "encryptionkey@0"
I20260929 16:32:09.519676 20106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:09.536204 20106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:09.540292 20106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:09.540603 20106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:09.541687 20106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:09.543097 20106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:09.543218 20106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:09.543370 20106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:09.543452 20106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:09.577792 20106 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45111
I20260929 16:32:09.577826 20233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45111 every 8 connection(s)
I20260929 16:32:09.579191 20106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:32:09.587775 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 20106
I20260929 16:32:09.587986 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:32:09.588449 6050 external_mini_cluster.cc:1428] Setting key 8a9221174b131e3a7e56d30c74ac808c
Sep 29 16:32:09 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699529, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:09.601145 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:09.581960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:56261 (local address 127.5.232.190:42935)
0929 16:32:09.582150 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:09.582156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:09.583348 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 16:32:09.585210 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:09.585216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:09.585309 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:09.585488 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:09.587034 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:09.587787 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:09.589805 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:09.590351 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:09.590457 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:09.594096 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:09.594105 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:09.594112 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:09.594162 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:09.597791 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:09.598581 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:09.598590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:09.598595 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:09.598774 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:09.599206 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:09.599212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:09.599216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:09.599612 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:09.599734 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:09.600653 (+ 919us) server_negotiation.cc:299] Negotiation successful
0929 16:32:09.600843 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:32:09.602850 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:09.582438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:56261)
0929 16:32:09.583176 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:09.583214 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:09.584910 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:09.585729 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:09.585738 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:09.586279 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:09.586850 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:09.586868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:09.587959 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:09.587964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:09.589663 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:09.589678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:09.590591 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:09.590601 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:09.590843 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:09.591556 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:09.591578 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:09.593837 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:09.597936 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:09.597944 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:09.597958 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:09.598442 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:09.598895 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:09.598899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:09.598901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:09.599087 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:09.599746 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:09.599757 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:09.600232 (+ 475us) client_negotiation.cc:769] Sending connection context
0929 16:32:09.600587 (+ 355us) client_negotiation.cc:240] Negotiation successful
0929 16:32:09.601270 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":284,"threads_started":1}
I20260929 16:32:09.604440 20234 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42935
I20260929 16:32:09.604781 20234 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:09.605603 20234 heartbeater.cc:511] Master 127.5.232.190:42935 requested a full tablet report, sending...
I20260929 16:32:09.607168 19911 ts_manager.cc:194] Registered new tserver with Master: 75486dabdb8b4ad188350506644cd676 (127.5.232.130:45111)
I20260929 16:32:09.608536 19911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:56261
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:32:09.642891 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42935
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:09.995687 20239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:09.995980 20239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:09.996147 20239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:10.003602 20239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:10.003736 20239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:10.003913 20239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:32:10.014213 20239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:10.015712 20239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:10.017323 20239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:10.031452 20244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:10.031387 20245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:10.031428 20247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:10.032615 20239 server_base.cc:1034] running on GCE node
I20260929 16:32:10.033283 20239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:10.034376 20239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:10.035660 20239 hybrid_clock.cc:648] HybridClock initialized: now 1790699530035596 us; error 61 us; skew 500 ppm
I20260929 16:32:10.036011 20239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:10 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699530, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:10.039482 20239 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:32:10.041181 20239 webserver.cc:460] Webserver started at http://127.5.232.131:44621/ using document root <none> and password file <none>
I20260929 16:32:10.042079 20239 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:10.042232 20239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:10.042572 20239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:10.045758 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4"
format_stamp: "Formatted at 2026-09-29 16:32:10 on dist-test-slave-dssx"
server_key: "e12776b9d86695969fa02945942d2dd8"
server_key_iv: "054765d50f8fd731acab7f8c091c3a6a"
server_key_version: "encryptionkey@0"
I20260929 16:32:10.046592 20239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4"
format_stamp: "Formatted at 2026-09-29 16:32:10 on dist-test-slave-dssx"
server_key: "e12776b9d86695969fa02945942d2dd8"
server_key_iv: "054765d50f8fd731acab7f8c091c3a6a"
server_key_version: "encryptionkey@0"
I20260929 16:32:10.052191 20239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:32:10.056133 20254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:10.057663 20239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:32:10.057969 20239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4"
format_stamp: "Formatted at 2026-09-29 16:32:10 on dist-test-slave-dssx"
server_key: "e12776b9d86695969fa02945942d2dd8"
server_key_iv: "054765d50f8fd731acab7f8c091c3a6a"
server_key_version: "encryptionkey@0"
I20260929 16:32:10.058198 20239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:10.086329 20239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:10.091046 20239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:10.091370 20239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:10.092370 20239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:10.093885 20239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:10.094002 20239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:10.094154 20239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:10.094228 20239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:10.096722 20100 heartbeater.cc:503] Master 127.5.232.190:42935 was elected leader, sending a full tablet report...
I20260929 16:32:10.129123 20239 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:46153
I20260929 16:32:10.129202 20366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:46153 every 8 connection(s)
I20260929 16:32:10.130487 20239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:32:10.138055 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 20239
I20260929 16:32:10.138268 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:32:10.138859 6050 external_mini_cluster.cc:1428] Setting key cb0d5c93f24cbfbcb58a036fbe0707f2
Sep 29 16:32:10 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699530, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:10.152220 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.133113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:47079 (local address 127.5.232.190:42935)
0929 16:32:10.133682 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.133689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.134680 (+ 991us) server_negotiation.cc:407] Connection header received
0929 16:32:10.136543 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.136554 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.136631 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.136741 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:10.138918 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.139976 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.142281 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.142719 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.142823 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.146424 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:10.146429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:10.146432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:10.146466 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:10.149289 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.150079 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.150083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.150085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.150210 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.150647 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.150650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.150652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.150926 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:10.151045 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.151756 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.151934 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":342,"threads_started":1}
I20260929 16:32:10.153254 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.133764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:47079)
0929 16:32:10.134500 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:10.134544 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:10.136219 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:10.137007 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:10.137017 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:10.137774 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:10.138679 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.138709 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.140159 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.140163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.142139 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.142152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.142925 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.142930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.143147 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.143807 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:10.143830 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:10.146200 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:10.149491 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:10.149504 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:10.149518 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:10.149953 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:10.150307 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:10.150310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:10.150312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:10.150527 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:10.151038 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:10.151043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:10.151383 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 16:32:10.151653 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 16:32:10.152069 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":272,"threads_started":1}
I20260929 16:32:10.154937 20367 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42935
I20260929 16:32:10.155313 20367 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:10.156427 20367 heartbeater.cc:511] Master 127.5.232.190:42935 requested a full tablet report, sending...
I20260929 16:32:10.157904 19911 ts_manager.cc:194] Registered new tserver with Master: b4cdfd7888934fa0ba8f2a8104cd28f4 (127.5.232.131:46153)
I20260929 16:32:10.158710 19911 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:47079
I20260929 16:32:10.167829 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:32:10.185114 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.172422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.5.232.190:42935)
0929 16:32:10.172677 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.172688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.173049 (+ 361us) server_negotiation.cc:407] Connection header received
0929 16:32:10.173198 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.173202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.173282 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.173414 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:10.174817 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.175555 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.176979 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.177442 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.177564 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.179937 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:10.179942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:10.179945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:10.179980 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:10.182623 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.183263 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.183266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.183268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.183390 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.183724 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.183727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.183729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.184050 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:10.184251 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.184559 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.184788 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:32:10.189596 19911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:32:10.196607 19911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:32:10.220667 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.209820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44406 (local address 127.5.232.129:40685)
0929 16:32:10.213083 (+ 3263us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.213095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.213126 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:32:10.213234 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.213243 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.213562 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.214338 (+ 776us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:10.214520 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.215529 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.218338 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.219343 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.219531 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.219787 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.220095 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.220289 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3092,"thread_start_us":2942,"threads_started":1}
I20260929 16:32:10.221423 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.209664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40685 (local address 127.0.0.1:44406)
0929 16:32:10.211093 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:10.211131 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:10.211309 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:10.213818 (+ 2509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:10.213823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:10.213848 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:10.214154 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.214162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.215678 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.215686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.218175 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.218185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.219566 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.219571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.219766 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.219822 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 16:32:10.220015 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 16:32:10.220383 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"mutex_wait_us":58,"thread_start_us":1034,"threads_started":1}
I20260929 16:32:10.225131 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.211872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.5.232.131:46153)
0929 16:32:10.216192 (+ 4320us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.216202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.216237 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:32:10.216341 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.216350 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.216642 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.216845 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:10.217953 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.219659 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.222828 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.224007 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.224183 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.224426 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.224616 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.224792 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4149,"thread_start_us":3985,"threads_started":1}
I20260929 16:32:10.225226 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.211547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:46153 (local address 127.0.0.1:49632)
0929 16:32:10.212502 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:10.212519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:10.212621 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:10.217128 (+ 4507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:10.217133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:10.217160 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:10.217727 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.217745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.219834 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.219839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.222648 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.222661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.224220 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.224227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.224400 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.224434 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:32:10.224571 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 16:32:10.224839 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":183,"spinlock_wait_cycles":2304,"thread_start_us":295,"threads_started":1}
I20260929 16:32:10.226116 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.210206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45111 (local address 127.0.0.1:36046)
0929 16:32:10.212107 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:10.212129 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:10.212269 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:10.217193 (+ 4924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:10.217198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:10.217226 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:10.217871 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.217889 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.219985 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.219992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.223409 (+ 3417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.223424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.225325 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.225333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.225536 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.225569 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:32:10.225641 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 16:32:10.225767 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1776,"mutex_wait_us":46,"thread_start_us":2027,"threads_started":1}
I20260929 16:32:10.226608 20383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.210848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.5.232.130:45111)
0929 16:32:10.216543 (+ 5695us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.216551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.216581 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:32:10.216693 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.216701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.216955 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.218107 (+ 1152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:10.218344 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.219836 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.223628 (+ 3792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.225092 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.225863 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.226046 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.226123 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.226280 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5548,"thread_start_us":5402,"threads_started":1}
I20260929 16:32:10.226101 20035 tablet_service.cc:1467] Processing CreateTablet for tablet ecacde770faa4ce6bdc58b700c65ed80 (DEFAULT_TABLE table=test-table [id=3c98d4391b124e41a84532e6e7602f8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:10.228365 20035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecacde770faa4ce6bdc58b700c65ed80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:10.229424 20302 tablet_service.cc:1467] Processing CreateTablet for tablet ecacde770faa4ce6bdc58b700c65ed80 (DEFAULT_TABLE table=test-table [id=3c98d4391b124e41a84532e6e7602f8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:10.230129 20169 tablet_service.cc:1467] Processing CreateTablet for tablet ecacde770faa4ce6bdc58b700c65ed80 (DEFAULT_TABLE table=test-table [id=3c98d4391b124e41a84532e6e7602f8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:10.230643 20302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecacde770faa4ce6bdc58b700c65ed80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:10.231469 20169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecacde770faa4ce6bdc58b700c65ed80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:10.243047 20386 tablet_bootstrap.cc:492] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: Bootstrap starting.
I20260929 16:32:10.246397 20388 tablet_bootstrap.cc:492] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: Bootstrap starting.
I20260929 16:32:10.246760 20386 tablet_bootstrap.cc:654] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:10.248279 20386 log.cc:824] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:10.248353 20387 tablet_bootstrap.cc:492] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: Bootstrap starting.
I20260929 16:32:10.250470 20388 tablet_bootstrap.cc:654] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:10.251688 20388 log.cc:824] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:10.252192 20387 tablet_bootstrap.cc:654] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:10.252326 20386 tablet_bootstrap.cc:492] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: No bootstrap required, opened a new log
I20260929 16:32:10.252632 20386 ts_tablet_manager.cc:1397] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260929 16:32:10.253561 20387 log.cc:824] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:10.254995 20388 tablet_bootstrap.cc:492] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: No bootstrap required, opened a new log
I20260929 16:32:10.255263 20388 ts_tablet_manager.cc:1397] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:32:10.256788 20387 tablet_bootstrap.cc:492] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: No bootstrap required, opened a new log
I20260929 16:32:10.257079 20387 ts_tablet_manager.cc:1397] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 16:32:10.262959 20388 raft_consensus.cc:348] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.263415 20388 raft_consensus.cc:374] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:10.263557 20388 raft_consensus.cc:729] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75486dabdb8b4ad188350506644cd676, State: Initialized, Role: FOLLOWER
I20260929 16:32:10.263293 20386 raft_consensus.cc:348] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.263731 20386 raft_consensus.cc:374] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:10.263832 20386 raft_consensus.cc:729] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4cdfd7888934fa0ba8f2a8104cd28f4, State: Initialized, Role: FOLLOWER
I20260929 16:32:10.264271 20388 consensus_queue.cc:260] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.264457 20386 consensus_queue.cc:260] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.265849 20367 heartbeater.cc:503] Master 127.5.232.190:42935 was elected leader, sending a full tablet report...
I20260929 16:32:10.266378 20386 ts_tablet_manager.cc:1428] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260929 16:32:10.267668 20387 raft_consensus.cc:348] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.268141 20387 raft_consensus.cc:374] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:10.268241 20387 raft_consensus.cc:729] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944912a5f3834f6db44b26e9ccb334f4, State: Initialized, Role: FOLLOWER
I20260929 16:32:10.268362 20234 heartbeater.cc:503] Master 127.5.232.190:42935 was elected leader, sending a full tablet report...
I20260929 16:32:10.269014 20388 ts_tablet_manager.cc:1428] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 16:32:10.268940 20387 consensus_queue.cc:260] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.272171 20387 ts_tablet_manager.cc:1428] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
W20260929 16:32:10.315800 20101 tablet.cc:2369] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:32:10.331364 20235 tablet.cc:2369] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:32:10.382140 20368 tablet.cc:2369] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:10.513043 20392 raft_consensus.cc:482] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:32:10.513409 20392 raft_consensus.cc:504] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.515270 20392 leader_election.cc:288] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685), b4cdfd7888934fa0ba8f2a8104cd28f4 (127.5.232.131:46153)
I20260929 16:32:10.515348 20393 raft_consensus.cc:482] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:32:10.515671 20393 raft_consensus.cc:504] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.524706 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.515747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:58301 (local address 127.5.232.129:40685)
0929 16:32:10.516000 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.516006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.516565 (+ 559us) server_negotiation.cc:407] Connection header received
0929 16:32:10.516917 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.516926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.517035 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.517115 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:10.518003 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.519177 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.521995 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.523383 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.523532 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.523621 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.524253 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.524419 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:32:10.524869 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.516908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:39495 (local address 127.5.232.131:46153)
0929 16:32:10.517087 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.517095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.517615 (+ 520us) server_negotiation.cc:407] Connection header received
0929 16:32:10.517819 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.517825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.517913 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.518006 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:10.518749 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.519856 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.522306 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.523649 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.523778 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.523853 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.524472 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.524606 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:32:10.525550 20393 leader_election.cc:288] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685), 75486dabdb8b4ad188350506644cd676 (127.5.232.130:45111)
I20260929 16:32:10.525720 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.516625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:46153 (local address 127.5.232.130:39495)
0929 16:32:10.517484 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:10.517503 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:10.517670 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:10.518164 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:10.518179 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:10.518203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:10.518575 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.518586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.520055 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.520060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.522123 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.522138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.524101 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.524106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.524293 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.524322 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:32:10.525093 (+ 771us) client_negotiation.cc:240] Negotiation successful
0929 16:32:10.525279 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":2,"thread_start_us":471,"threads_started":1}
I20260929 16:32:10.526917 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.515682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40685 (local address 127.5.232.130:58301)
0929 16:32:10.516463 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:10.516485 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:10.516702 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:10.517338 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:10.517341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:10.517363 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:10.517825 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.517839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.519358 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.519364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.521796 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.521812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.523841 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.523846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.524037 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.524086 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 16:32:10.526411 (+ 2325us) client_negotiation.cc:240] Negotiation successful
0929 16:32:10.526560 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":6,"thread_start_us":439,"threads_started":1}
I20260929 16:32:10.526840 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecacde770faa4ce6bdc58b700c65ed80" candidate_uuid: "75486dabdb8b4ad188350506644cd676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" is_pre_election: true
I20260929 16:32:10.527372 20322 raft_consensus.cc:2454] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75486dabdb8b4ad188350506644cd676 in term 0.
I20260929 16:32:10.528093 20055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecacde770faa4ce6bdc58b700c65ed80" candidate_uuid: "75486dabdb8b4ad188350506644cd676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944912a5f3834f6db44b26e9ccb334f4" is_pre_election: true
I20260929 16:32:10.528322 20123 leader_election.cc:302] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75486dabdb8b4ad188350506644cd676, b4cdfd7888934fa0ba8f2a8104cd28f4; no voters:
I20260929 16:32:10.528718 20055 raft_consensus.cc:2454] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75486dabdb8b4ad188350506644cd676 in term 0.
I20260929 16:32:10.529230 20392 raft_consensus.cc:2781] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:32:10.529428 20392 raft_consensus.cc:482] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:32:10.529537 20392 raft_consensus.cc:3037] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:10.532608 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.518062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40685 (local address 127.5.232.131:49081)
0929 16:32:10.518431 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:10.518455 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:10.518585 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:10.525519 (+ 6934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:10.525524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:10.525559 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:10.525771 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.525786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.527772 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.527777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.529858 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.529875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.531738 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.531745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.531984 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.532024 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:32:10.532135 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:32:10.532299 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260929 16:32:10.532666 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.518098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:49081 (local address 127.5.232.129:40685)
0929 16:32:10.524981 (+ 6883us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.524987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.525007 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:32:10.525069 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.525075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.525157 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.525250 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:10.525948 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.527057 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.530061 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.531375 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.531555 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.531630 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.532201 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.532358 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6776,"thread_start_us":9285,"threads_started":1}
I20260929 16:32:10.533082 20392 raft_consensus.cc:504] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.533599 20055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecacde770faa4ce6bdc58b700c65ed80" candidate_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944912a5f3834f6db44b26e9ccb334f4" is_pre_election: true
I20260929 16:32:10.533841 20055 raft_consensus.cc:2454] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4cdfd7888934fa0ba8f2a8104cd28f4 in term 0.
I20260929 16:32:10.534297 20055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecacde770faa4ce6bdc58b700c65ed80" candidate_uuid: "75486dabdb8b4ad188350506644cd676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944912a5f3834f6db44b26e9ccb334f4"
I20260929 16:32:10.534504 20055 raft_consensus.cc:3037] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:10.534504 20256 leader_election.cc:302] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944912a5f3834f6db44b26e9ccb334f4, b4cdfd7888934fa0ba8f2a8104cd28f4; no voters:
I20260929 16:32:10.534776 20392 leader_election.cc:288] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [CANDIDATE]: Term 1 election: Requested vote from peers 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685), b4cdfd7888934fa0ba8f2a8104cd28f4 (127.5.232.131:46153)
I20260929 16:32:10.535118 20393 raft_consensus.cc:2781] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:32:10.535311 20393 raft_consensus.cc:482] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:32:10.535419 20393 raft_consensus.cc:3037] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:10.536212 20383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.519829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:52599 (local address 127.5.232.130:45111)
0929 16:32:10.520043 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.520052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.529369 (+ 9317us) server_negotiation.cc:407] Connection header received
0929 16:32:10.529662 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.529667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.529740 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.529806 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:10.530585 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.531351 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.533777 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.535018 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.535184 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.535262 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.535810 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.535959 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 16:32:10.536262 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.519197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45111 (local address 127.5.232.131:52599)
0929 16:32:10.529278 (+ 10081us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:10.529298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:10.529506 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:10.530011 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:10.530015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:10.530046 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:10.530439 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.530453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.531511 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.531519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.533576 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:10.533590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.535400 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.535406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:10.535593 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.535631 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:32:10.535746 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:32:10.535893 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9918,"thread_start_us":9739,"threads_started":1}
I20260929 16:32:10.536893 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecacde770faa4ce6bdc58b700c65ed80" candidate_uuid: "75486dabdb8b4ad188350506644cd676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4"
I20260929 16:32:10.537245 20189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecacde770faa4ce6bdc58b700c65ed80" candidate_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75486dabdb8b4ad188350506644cd676" is_pre_election: true
I20260929 16:32:10.537778 20189 raft_consensus.cc:2379] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b4cdfd7888934fa0ba8f2a8104cd28f4 in current term 1: Already voted for candidate 75486dabdb8b4ad188350506644cd676 in this term.
I20260929 16:32:10.537894 20055 raft_consensus.cc:2454] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75486dabdb8b4ad188350506644cd676 in term 1.
I20260929 16:32:10.538360 20123 leader_election.cc:302] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75486dabdb8b4ad188350506644cd676, 944912a5f3834f6db44b26e9ccb334f4; no voters:
I20260929 16:32:10.538693 20393 raft_consensus.cc:504] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.539187 20322 raft_consensus.cc:2379] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 75486dabdb8b4ad188350506644cd676 in current term 1: Already voted for candidate b4cdfd7888934fa0ba8f2a8104cd28f4 in this term.
I20260929 16:32:10.539455 20392 raft_consensus.cc:2781] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:32:10.539588 20393 leader_election.cc:288] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [CANDIDATE]: Term 1 election: Requested vote from peers 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685), 75486dabdb8b4ad188350506644cd676 (127.5.232.130:45111)
I20260929 16:32:10.540283 20055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecacde770faa4ce6bdc58b700c65ed80" candidate_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944912a5f3834f6db44b26e9ccb334f4"
I20260929 16:32:10.540604 20055 raft_consensus.cc:2379] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b4cdfd7888934fa0ba8f2a8104cd28f4 in current term 1: Already voted for candidate 75486dabdb8b4ad188350506644cd676 in this term.
I20260929 16:32:10.540612 20189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecacde770faa4ce6bdc58b700c65ed80" candidate_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75486dabdb8b4ad188350506644cd676"
I20260929 16:32:10.541265 20392 raft_consensus.cc:686] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 1 LEADER]: Becoming Leader. State: Replica: 75486dabdb8b4ad188350506644cd676, State: Running, Role: LEADER
I20260929 16:32:10.542037 20392 consensus_queue.cc:237] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:10.545853 20255 leader_election.cc:302] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4cdfd7888934fa0ba8f2a8104cd28f4; no voters: 75486dabdb8b4ad188350506644cd676, 944912a5f3834f6db44b26e9ccb334f4
I20260929 16:32:10.546180 20393 raft_consensus.cc:2726] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 16:32:10.547509 19909 catalog_manager.cc:5495] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75486dabdb8b4ad188350506644cd676 (127.5.232.130). New cstate: current_term: 1 leader_uuid: "75486dabdb8b4ad188350506644cd676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } health_report { overall_health: HEALTHY } } }
I20260929 16:32:10.591240 20383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.584861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.5.232.130:45111)
0929 16:32:10.585036 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.585043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.585117 (+ 74us) server_negotiation.cc:407] Connection header received
0929 16:32:10.585309 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.585316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.585451 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.585615 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:32:10.585957 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.587144 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.588611 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.589261 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.589416 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.589970 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:32:10.590608 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:32:10.590771 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.590820 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.590922 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:32:10.600149 20055 raft_consensus.cc:1261] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 1 FOLLOWER]: Refusing update from remote peer 75486dabdb8b4ad188350506644cd676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:10.600163 20322 raft_consensus.cc:1261] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 FOLLOWER]: Refusing update from remote peer 75486dabdb8b4ad188350506644cd676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:10.601084 20392 consensus_queue.cc:1035] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:10.601485 20399 consensus_queue.cc:1035] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:10.629765 19911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:10.630085 19911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:10.634037 19911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Sending DeleteTablet for 3 replicas of tablet ecacde770faa4ce6bdc58b700c65ed80
I20260929 16:32:10.635416 20035 tablet_service.cc:1514] Processing DeleteTablet for tablet ecacde770faa4ce6bdc58b700c65ed80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:10 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:44406
I20260929 16:32:10.635421 20169 tablet_service.cc:1514] Processing DeleteTablet for tablet ecacde770faa4ce6bdc58b700c65ed80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:10 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:36046
I20260929 16:32:10.635494 20302 tablet_service.cc:1514] Processing DeleteTablet for tablet ecacde770faa4ce6bdc58b700c65ed80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:10 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:49632
I20260929 16:32:10.636639 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 19972
I20260929 16:32:10.647908 20413 tablet_replica.cc:330] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: stopping tablet replica
I20260929 16:32:10.649039 20412 tablet_replica.cc:330] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: stopping tablet replica
I20260929 16:32:10.649304 20414 tablet_replica.cc:330] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: stopping tablet replica
I20260929 16:32:10.649554 20412 raft_consensus.cc:2229] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:32:10.649572 20413 raft_consensus.cc:2229] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:32:10.649818 20414 raft_consensus.cc:2229] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:32:10.650133 20412 raft_consensus.cc:2258] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:10.650208 20413 raft_consensus.cc:2258] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:10.650636 20414 raft_consensus.cc:2258] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:10.652734 20413 ts_tablet_manager.cc:1905] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:10.652751 20412 ts_tablet_manager.cc:1905] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:10.657135 20414 ts_tablet_manager.cc:1905] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:10.662469 20413 ts_tablet_manager.cc:1918] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:10.662572 20412 ts_tablet_manager.cc:1918] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:10.662676 20413 log.cc:1196] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/ecacde770faa4ce6bdc58b700c65ed80
I20260929 16:32:10.662763 20412 log.cc:1196] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/ecacde770faa4ce6bdc58b700c65ed80
I20260929 16:32:10.663318 20413 ts_tablet_manager.cc:1939] T ecacde770faa4ce6bdc58b700c65ed80 P 944912a5f3834f6db44b26e9ccb334f4: Deleting consensus metadata
I20260929 16:32:10.663336 20412 ts_tablet_manager.cc:1939] T ecacde770faa4ce6bdc58b700c65ed80 P b4cdfd7888934fa0ba8f2a8104cd28f4: Deleting consensus metadata
Sep 29 16:32:10 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:32:10.665357 19896 catalog_manager.cc:4841] TS b4cdfd7888934fa0ba8f2a8104cd28f4 (127.5.232.131:46153): tablet ecacde770faa4ce6bdc58b700c65ed80 (table test-table [id=3c98d4391b124e41a84532e6e7602f8f]) successfully deleted
I20260929 16:32:10.665797 19896 catalog_manager.cc:4841] TS 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685): tablet ecacde770faa4ce6bdc58b700c65ed80 (table test-table [id=3c98d4391b124e41a84532e6e7602f8f]) successfully deleted
I20260929 16:32:10.668650 20414 ts_tablet_manager.cc:1918] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:10.668870 20414 log.cc:1196] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/ecacde770faa4ce6bdc58b700c65ed80
I20260929 16:32:10.669425 20414 ts_tablet_manager.cc:1939] T ecacde770faa4ce6bdc58b700c65ed80 P 75486dabdb8b4ad188350506644cd676: Deleting consensus metadata
I20260929 16:32:10.671334 19895 catalog_manager.cc:4841] TS 75486dabdb8b4ad188350506644cd676 (127.5.232.130:45111): tablet ecacde770faa4ce6bdc58b700c65ed80 (table test-table [id=3c98d4391b124e41a84532e6e7602f8f]) successfully deleted
I20260929 16:32:10.673661 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.637671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.5.232.129:40685)
0929 16:32:10.637915 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.637924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.648255 (+ 10331us) server_negotiation.cc:407] Connection header received
0929 16:32:10.650600 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.650609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.650750 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.650855 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:10.653752 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.656097 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.658928 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.659651 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.659778 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.666107 (+ 6329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:10.666123 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:10.666143 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:10.666202 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:10.670292 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.671374 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.671385 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.671390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.671572 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.671972 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.671977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.671981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.672476 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:10.672803 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.673036 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.673281 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:32:10.674396 20095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:10.767891 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 19972
I20260929 16:32:10.787472 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 20106
Sep 29 16:32:10 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:32:10.805533 20383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.788310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.5.232.130:45111)
0929 16:32:10.788522 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.788528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.788551 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:32:10.788683 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.788689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.788792 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.788905 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:10.790277 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.791421 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.792704 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.793459 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.793595 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.798126 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:10.798148 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:10.798170 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:10.798229 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:10.802375 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.803183 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.803194 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.803198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.803408 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.803823 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.803829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.803832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.804368 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:10.804691 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.804865 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.805122 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:32:10.806267 20229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:10.925268 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 20106
I20260929 16:32:10.944434 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 20239
Sep 29 16:32:10 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:32:10.960661 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:10.945278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.5.232.131:46153)
0929 16:32:10.945492 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:10.945505 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:10.945600 (+ 95us) server_negotiation.cc:407] Connection header received
0929 16:32:10.945826 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:10.945830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:10.945896 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:10.946021 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:10.947446 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.948203 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.949550 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:10.949955 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:10.950097 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:10.954832 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:10.954849 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:10.954866 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:10.954952 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:10.958068 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.958870 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.958878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.958881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.959043 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:10.959370 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:10.959373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:10.959375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:10.959730 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:10.959981 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:10.960180 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 16:32:10.960350 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:32:10.961355 20362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:11.055291 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 20239
I20260929 16:32:11.073865 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 19879
I20260929 16:32:11.075690 19941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:11.195215 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 19879
I20260929 16:32:11.217237 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42935
--webserver_interface=127.5.232.190
--webserver_port=35541
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:11.563087 20420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:11.563462 20420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:11.563647 20420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:11.571463 20420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:11.571635 20420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:11.571749 20420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:11.571851 20420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:11.571940 20420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:11.582644 20420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=35541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:11.584084 20420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:11.585551 20420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:11.609985 20427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:11.610101 20429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:11.609980 20426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:11.610929 20420 server_base.cc:1034] running on GCE node
I20260929 16:32:11.611889 20420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:11.613598 20420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:11.614838 20420 hybrid_clock.cc:648] HybridClock initialized: now 1790699531614789 us; error 43 us; skew 500 ppm
I20260929 16:32:11.615346 20420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:11 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699531, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:11.619189 20420 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:32:11.621088 20420 webserver.cc:460] Webserver started at http://127.5.232.190:35541/ using document root <none> and password file <none>
I20260929 16:32:11.622011 20420 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:11.622187 20420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:11.628576 20420 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 16:32:11.631850 20436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:11.633518 20420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:32:11.633728 20420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "e3342ffbcf0d421eb102d6a3cee2d929"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "4c52e4caf8bf994cd9e492e6e4a52da8"
server_key_iv: "8025bebd3de18f9c7e231f7a39547649"
server_key_version: "encryptionkey@0"
I20260929 16:32:11.634905 20420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:11.648998 20420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:11.653893 20420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:11.654215 20420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:11.674589 20420 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:42935
I20260929 16:32:11.674633 20487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:42935 every 8 connection(s)
I20260929 16:32:11.676275 20420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:32:11.681870 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 20420
I20260929 16:32:11.682934 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:40685
--local_ip_for_outbound_sockets=127.5.232.129
--tserver_master_addrs=127.5.232.190:42935
--webserver_port=41587
--webserver_interface=127.5.232.129
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:32:11.685693 20488 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:32:11.691562 20488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Bootstrap starting.
I20260929 16:32:11.712435 20488 log.cc:824] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:11.727905 20488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:32:11.728520 20488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Bootstrap complete.
I20260929 16:32:11.738643 20488 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:11.739714 20488 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3342ffbcf0d421eb102d6a3cee2d929, State: Initialized, Role: FOLLOWER
I20260929 16:32:11.740429 20488 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:11.740691 20488 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:32:11.740818 20488 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:32:11.741014 20488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:32:11.745186 20488 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:11.745779 20488 leader_election.cc:302] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3342ffbcf0d421eb102d6a3cee2d929; no voters:
I20260929 16:32:11.746528 20488 leader_election.cc:288] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:32:11.746681 20492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:32:11.748001 20492 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 2 LEADER]: Becoming Leader. State: Replica: e3342ffbcf0d421eb102d6a3cee2d929, State: Running, Role: LEADER
I20260929 16:32:11.748634 20492 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:11.749046 20488 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:32:11.751379 20493 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } } }
I20260929 16:32:11.751875 20493 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:11.752477 20494 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3342ffbcf0d421eb102d6a3cee2d929. Latest consensus state: current_term: 2 leader_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } } }
I20260929 16:32:11.752811 20494 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:11.758776 20498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:32:11.767649 20498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:32:11.769318 20498 catalog_manager.cc:1259] Loaded cluster ID: 2bcd1b30b116434a9f0b660fef3f7480
I20260929 16:32:11.769512 20498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:32:11.772918 20498 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 16:32:11.791973 20498 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 16:32:11.801352 20498 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:32:11.803347 20498 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Loaded TSK: 0
I20260929 16:32:11.804855 20498 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:32:12.049778 20490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:12.050076 20490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:12.050261 20490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:12.057893 20490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:12.058008 20490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:12.058187 20490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:32:12.068341 20490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:40685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=41587
--tserver_master_addrs=127.5.232.190:42935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:12.069871 20490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:12.071281 20490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:12.086244 20516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:12.086764 20515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:12.086841 20518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:12.087225 20490 server_base.cc:1034] running on GCE node
I20260929 16:32:12.087911 20490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:12.088966 20490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:12.090242 20490 hybrid_clock.cc:648] HybridClock initialized: now 1790699532090176 us; error 60 us; skew 500 ppm
I20260929 16:32:12.090612 20490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:12 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699532, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:12.093957 20490 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:32:12.095652 20490 webserver.cc:460] Webserver started at http://127.5.232.129:41587/ using document root <none> and password file <none>
I20260929 16:32:12.096376 20490 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:12.096521 20490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:12.102334 20490 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:32:12.105276 20525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:12.106848 20490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:32:12.107021 20490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "944912a5f3834f6db44b26e9ccb334f4"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "fe572d280e2dd3f667986078fbbf8259"
server_key_iv: "d09d10386db0882bf21703e041f4ef2a"
server_key_version: "encryptionkey@0"
I20260929 16:32:12.108028 20490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:12.132052 20490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:12.136839 20490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:12.137153 20490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:12.138314 20490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:12.139946 20490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:12.140048 20490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:12.140209 20490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:12.140304 20490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:12.174757 20490 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:40685
I20260929 16:32:12.174852 20637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:40685 every 8 connection(s)
I20260929 16:32:12.176293 20490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:32:12.184121 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 20490
I20260929 16:32:12.184796 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:45111
--local_ip_for_outbound_sockets=127.5.232.130
--tserver_master_addrs=127.5.232.190:42935
--webserver_port=42403
--webserver_interface=127.5.232.130
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:32:12 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699532, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:12.202440 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:12.179313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:33021 (local address 127.5.232.190:42935)
0929 16:32:12.179901 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:12.179914 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:12.180761 (+ 847us) server_negotiation.cc:407] Connection header received
0929 16:32:12.182993 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:12.183007 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:12.183380 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:12.183813 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:12.185629 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:12.186899 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:12.188553 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:12.189263 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:12.189515 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:12.194159 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:12.194168 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:12.194186 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:12.194230 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:12.197441 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:12.198341 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:12.198349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:12.198355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:12.198535 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:12.199026 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:12.199030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:12.199033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:12.199550 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:12.199796 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:12.200646 (+ 850us) server_negotiation.cc:299] Negotiation successful
0929 16:32:12.201042 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
I20260929 16:32:12.202440 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:12.179607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:33021)
0929 16:32:12.180561 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:12.180615 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:12.182464 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:12.183993 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:12.184006 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:12.184659 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:12.185314 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:12.185334 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:12.187099 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:12.187104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:12.188385 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:12.188398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:12.189552 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:12.189560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:12.189852 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:12.190673 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:12.190706 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:12.193819 (+ 3113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:12.197635 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:12.197645 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:12.197663 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:12.198159 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:12.198670 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:12.198675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:12.198679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:12.198887 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:12.199730 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:12.199738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:12.200119 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 16:32:12.200405 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 16:32:12.201017 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":388,"threads_started":1}
I20260929 16:32:12.206188 20638 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42935
I20260929 16:32:12.206523 20638 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:12.207291 20638 heartbeater.cc:511] Master 127.5.232.190:42935 requested a full tablet report, sending...
I20260929 16:32:12.209911 20453 ts_manager.cc:194] Registered new tserver with Master: 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685)
I20260929 16:32:12.212183 20453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:33021
W20260929 16:32:12.521446 20642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:12.521752 20642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:12.521849 20642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:12.529086 20642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:12.529206 20642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:12.529322 20642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:32:12.539171 20642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:45111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=42403
--tserver_master_addrs=127.5.232.190:42935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:12.540548 20642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:12.541877 20642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:12.555066 20648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:12.555105 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:12.555002 20647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:12.555888 20642 server_base.cc:1034] running on GCE node
I20260929 16:32:12.556595 20642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:12.557622 20642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:12.558858 20642 hybrid_clock.cc:648] HybridClock initialized: now 1790699532558818 us; error 55 us; skew 500 ppm
I20260929 16:32:12.559227 20642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:12 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699532, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:12.562407 20642 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:32:12.563932 20642 webserver.cc:460] Webserver started at http://127.5.232.130:42403/ using document root <none> and password file <none>
I20260929 16:32:12.564608 20642 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:12.564718 20642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:12.570389 20642 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:32:12.573438 20657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:12.574946 20642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:32:12.575098 20642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "75486dabdb8b4ad188350506644cd676"
format_stamp: "Formatted at 2026-09-29 16:32:09 on dist-test-slave-dssx"
server_key: "a0b80b3d61393410547cf9265e86aaa6"
server_key_iv: "6a46d744e67c464f2248e3b489b19a3e"
server_key_version: "encryptionkey@0"
I20260929 16:32:12.576076 20642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:12.607198 20642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:12.611681 20642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:12.611919 20642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:12.612963 20642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:12.614543 20642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:12.614640 20642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:12.614804 20642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:12.614892 20642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:12.648947 20642 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45111
I20260929 16:32:12.649094 20769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45111 every 8 connection(s)
I20260929 16:32:12.650475 20642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:32:12.659615 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 20642
I20260929 16:32:12.660391 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:46153
--local_ip_for_outbound_sockets=127.5.232.131
--tserver_master_addrs=127.5.232.190:42935
--webserver_port=44621
--webserver_interface=127.5.232.131
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:32:12 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699532, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:12.673997 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:12.653182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:35327 (local address 127.5.232.190:42935)
0929 16:32:12.653439 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:12.653446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:12.654664 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 16:32:12.656520 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:12.656525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:12.656598 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:12.656708 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:12.658333 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:12.659050 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:12.661681 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:12.662117 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:12.662232 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:12.667014 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:12.667019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:12.667023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:12.667062 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:12.670066 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:12.671175 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:12.671180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:12.671182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:12.671322 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:12.671849 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:12.671852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:12.671854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:12.672125 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:12.672247 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:12.673231 (+ 984us) server_negotiation.cc:299] Negotiation successful
0929 16:32:12.673526 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:32:12.675730 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:12.653736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:35327)
0929 16:32:12.654472 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:12.654525 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:12.656214 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:12.656921 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:12.656929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:12.657547 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:12.658140 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:12.658169 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:12.659235 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:12.659242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:12.661474 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:12.661494 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:12.662368 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:12.662377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:12.662714 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:12.663475 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:12.663511 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:12.666728 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:12.670246 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:12.670260 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:12.670284 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:12.671025 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:12.671449 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:12.671457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:12.671464 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:12.671731 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:12.672280 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:12.672292 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:12.672798 (+ 506us) client_negotiation.cc:769] Sending connection context
0929 16:32:12.673115 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 16:32:12.673768 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260929 16:32:12.677373 20770 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42935
I20260929 16:32:12.677762 20770 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:12.678696 20770 heartbeater.cc:511] Master 127.5.232.190:42935 requested a full tablet report, sending...
I20260929 16:32:12.680540 20453 ts_manager.cc:194] Registered new tserver with Master: 75486dabdb8b4ad188350506644cd676 (127.5.232.130:45111)
I20260929 16:32:12.682049 20453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:35327
W20260929 16:32:13.004666 20773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:13.004963 20773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:13.005131 20773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:13.012691 20773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:13.012811 20773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:13.013003 20773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:32:13.023169 20773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:46153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=44621
--tserver_master_addrs=127.5.232.190:42935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:13.024559 20773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:13.025997 20773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:13.039783 20779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:13.039762 20781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:13.040448 20778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:13.041031 20773 server_base.cc:1034] running on GCE node
I20260929 16:32:13.041745 20773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:13.042739 20773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:13.043947 20773 hybrid_clock.cc:648] HybridClock initialized: now 1790699533043895 us; error 39 us; skew 500 ppm
I20260929 16:32:13.044291 20773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:13 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699533, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:13.047505 20773 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:32:13.048987 20773 webserver.cc:460] Webserver started at http://127.5.232.131:44621/ using document root <none> and password file <none>
I20260929 16:32:13.049791 20773 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:13.049934 20773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:13.055495 20773 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:32:13.058416 20788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:13.059943 20773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:32:13.060128 20773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4"
format_stamp: "Formatted at 2026-09-29 16:32:10 on dist-test-slave-dssx"
server_key: "e12776b9d86695969fa02945942d2dd8"
server_key_iv: "054765d50f8fd731acab7f8c091c3a6a"
server_key_version: "encryptionkey@0"
I20260929 16:32:13.061146 20773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:13.091076 20773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:13.095440 20773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:13.095755 20773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:13.096913 20773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:13.098385 20773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:13.098474 20773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:13.098637 20773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:13.098730 20773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:13.132977 20773 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:46153
I20260929 16:32:13.133061 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:46153 every 8 connection(s)
I20260929 16:32:13.134585 20773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:32:13.142066 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 20773
Sep 29 16:32:13 dist-test-slave-dssx krb5kdc[19866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699533, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:13.157034 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.137299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:39743 (local address 127.5.232.190:42935)
0929 16:32:13.137507 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.137512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.138743 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 16:32:13.140723 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.140729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.140830 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.140947 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:13.143046 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.144106 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.145804 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.146285 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.146432 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.150377 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:13.150382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:13.150385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:13.150416 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:13.153840 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.154628 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.154633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.154637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.154770 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.155232 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.155237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.155240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.155539 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:13.155708 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.156443 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.156680 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:32:13.157661 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.143150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.5.232.190:42935)
0929 16:32:13.143986 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.143991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.144007 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:32:13.144054 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.144057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.144112 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.144248 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:13.145790 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.146589 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.147978 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.148408 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.148556 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.151581 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:13.151586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:13.151589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:13.151635 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:13.154593 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.155321 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.155328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.155336 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.155452 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.155848 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.155853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.155857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.156118 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:13.156386 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.156732 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.156960 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":574,"threads_started":1}
I20260929 16:32:13.158286 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.137784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:39743)
0929 16:32:13.138585 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:13.138630 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:13.140433 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:13.141214 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:13.141223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:13.142021 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:13.142866 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.142893 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.144274 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.144279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.145656 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.145669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.146545 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.146553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.146804 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.147534 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:13.147560 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:13.150117 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:13.153991 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:13.154002 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:13.154021 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:13.154484 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:13.154885 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:13.154891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:13.154894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:13.155102 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:13.155683 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:13.155691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:13.156064 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 16:32:13.156343 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 16:32:13.156920 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":302,"threads_started":1}
I20260929 16:32:13.159868 20901 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42935
I20260929 16:32:13.160228 20901 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:13.160979 20901 heartbeater.cc:511] Master 127.5.232.190:42935 requested a full tablet report, sending...
I20260929 16:32:13.162525 20452 ts_manager.cc:194] Registered new tserver with Master: b4cdfd7888934fa0ba8f2a8104cd28f4 (127.5.232.131:46153)
I20260929 16:32:13.163360 20452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:39743
I20260929 16:32:13.173130 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:32:13.181577 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.174189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.5.232.190:42935)
0929 16:32:13.174435 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.174444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.174832 (+ 388us) server_negotiation.cc:407] Connection header received
0929 16:32:13.175100 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.175105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.175263 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.175420 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:32:13.175874 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.176644 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.178945 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.179504 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.179610 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.180272 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:32:13.180851 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:32:13.181026 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.181162 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.181238 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 16:32:13.182593 20452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:32:13.190408 20452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:32:13.216049 20638 heartbeater.cc:503] Master 127.5.232.190:42935 was elected leader, sending a full tablet report...
I20260929 16:32:13.217423 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.203353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.5.232.131:46153)
0929 16:32:13.204263 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.204277 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.205077 (+ 800us) server_negotiation.cc:407] Connection header received
0929 16:32:13.205546 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.205559 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.205848 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.206029 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:13.207339 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.209362 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.212958 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.214983 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.215127 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.215446 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.216656 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.216910 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":451,"threads_started":1}
I20260929 16:32:13.218995 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.203234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:46153 (local address 127.0.0.1:49646)
0929 16:32:13.204927 (+ 1693us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:13.204964 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:13.205229 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:13.206456 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:13.206472 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:13.206529 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:13.207081 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.207098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.209697 (+ 2599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.209705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.212706 (+ 3001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.212732 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.215746 (+ 3014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.215756 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.216053 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.216129 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 16:32:13.218196 (+ 2067us) client_negotiation.cc:240] Negotiation successful
0929 16:32:13.218520 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"thread_start_us":1290,"threads_started":1}
I20260929 16:32:13.226789 20836 tablet_service.cc:1467] Processing CreateTablet for tablet d36395d44e4e480eab4e5df24bb6c348 (DEFAULT_TABLE table=test-table [id=bcc20aa37e3f43bd8e4c0b3bbdc4ad6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:13.228627 20836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36395d44e4e480eab4e5df24bb6c348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:13.231374 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.205453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40685 (local address 127.0.0.1:44426)
0929 16:32:13.222327 (+ 16874us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:13.222345 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:13.222484 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:13.223540 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:13.223545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:13.223577 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:13.224021 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.224034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.226064 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.226071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.228121 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.228142 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.230479 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.230487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.230686 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.230722 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:32:13.230825 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:32:13.231016 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16748,"thread_start_us":16597,"threads_started":1}
I20260929 16:32:13.232180 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.205915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.5.232.129:40685)
0929 16:32:13.221865 (+ 15950us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.221876 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.222625 (+ 749us) server_negotiation.cc:407] Connection header received
0929 16:32:13.222743 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.222750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.223042 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.223256 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:13.224243 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.225847 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.228346 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.229877 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.230044 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.230306 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.231659 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.231846 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15763,"thread_start_us":15596,"threads_started":1}
I20260929 16:32:13.240041 20573 tablet_service.cc:1467] Processing CreateTablet for tablet d36395d44e4e480eab4e5df24bb6c348 (DEFAULT_TABLE table=test-table [id=bcc20aa37e3f43bd8e4c0b3bbdc4ad6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:13.241750 20573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36395d44e4e480eab4e5df24bb6c348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:13.247529 20917 tablet_bootstrap.cc:492] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: Bootstrap starting.
I20260929 16:32:13.249655 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.203942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45111 (local address 127.0.0.1:36080)
0929 16:32:13.219516 (+ 15574us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:13.219540 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:13.219682 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:13.238551 (+ 18869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:13.238560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:13.238607 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:13.238886 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.238904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.242596 (+ 3692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.242602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.244792 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.244827 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.248662 (+ 3835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.248671 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.248880 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.248922 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:32:13.249007 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:32:13.249275 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15430,"thread_start_us":17459,"threads_started":1}
I20260929 16:32:13.250937 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.204520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.5.232.130:45111)
0929 16:32:13.237759 (+ 33239us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.237769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.237804 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:32:13.237914 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.237925 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.238214 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.239104 (+ 890us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:13.239452 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.242370 (+ 2918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.245090 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.248408 (+ 3318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.250049 (+ 1641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.250311 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.250393 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.250586 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33052,"thread_start_us":32886,"threads_started":1}
I20260929 16:32:13.252719 20917 tablet_bootstrap.cc:654] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:13.254261 20917 log.cc:824] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:13.255304 20705 tablet_service.cc:1467] Processing CreateTablet for tablet d36395d44e4e480eab4e5df24bb6c348 (DEFAULT_TABLE table=test-table [id=bcc20aa37e3f43bd8e4c0b3bbdc4ad6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:13.258159 20917 tablet_bootstrap.cc:492] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: No bootstrap required, opened a new log
I20260929 16:32:13.258508 20917 ts_tablet_manager.cc:1397] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 16:32:13.259426 20705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36395d44e4e480eab4e5df24bb6c348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:13.263109 20919 tablet_bootstrap.cc:492] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: Bootstrap starting.
I20260929 16:32:13.267277 20919 tablet_bootstrap.cc:654] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:13.268893 20919 log.cc:824] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:13.269877 20917 raft_consensus.cc:348] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:13.270440 20917 raft_consensus.cc:374] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:13.270548 20917 raft_consensus.cc:729] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4cdfd7888934fa0ba8f2a8104cd28f4, State: Initialized, Role: FOLLOWER
I20260929 16:32:13.271363 20917 consensus_queue.cc:260] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:13.272782 20901 heartbeater.cc:503] Master 127.5.232.190:42935 was elected leader, sending a full tablet report...
I20260929 16:32:13.273348 20917 ts_tablet_manager.cc:1428] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260929 16:32:13.277527 20919 tablet_bootstrap.cc:492] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: No bootstrap required, opened a new log
I20260929 16:32:13.277851 20919 ts_tablet_manager.cc:1397] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260929 16:32:13.278098 20922 tablet_bootstrap.cc:492] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: Bootstrap starting.
I20260929 16:32:13.284816 20922 tablet_bootstrap.cc:654] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:13.286597 20922 log.cc:824] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:13.289251 20919 raft_consensus.cc:348] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:13.289799 20919 raft_consensus.cc:374] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:13.289907 20919 raft_consensus.cc:729] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944912a5f3834f6db44b26e9ccb334f4, State: Initialized, Role: FOLLOWER
I20260929 16:32:13.290557 20922 tablet_bootstrap.cc:492] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: No bootstrap required, opened a new log
I20260929 16:32:13.290591 20919 consensus_queue.cc:260] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:13.290903 20922 ts_tablet_manager.cc:1397] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260929 16:32:13.292443 20919 ts_tablet_manager.cc:1428] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 16:32:13.302546 20922 raft_consensus.cc:348] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:13.302945 20922 raft_consensus.cc:374] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:13.303076 20922 raft_consensus.cc:729] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75486dabdb8b4ad188350506644cd676, State: Initialized, Role: FOLLOWER
I20260929 16:32:13.303704 20922 consensus_queue.cc:260] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:13.304858 20770 heartbeater.cc:503] Master 127.5.232.190:42935 was elected leader, sending a full tablet report...
I20260929 16:32:13.305337 20922 ts_tablet_manager.cc:1428] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260929 16:32:13.333626 20921 raft_consensus.cc:482] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:32:13.333966 20921 raft_consensus.cc:504] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:13.335712 20921 leader_election.cc:288] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685), 75486dabdb8b4ad188350506644cd676 (127.5.232.130:45111)
I20260929 16:32:13.342151 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.336318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:58753 (local address 127.5.232.130:45111)
0929 16:32:13.336600 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.336610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.336634 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:32:13.336714 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.336719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.336801 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.336909 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:13.337460 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.338174 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.340450 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.341241 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.341434 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.341509 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.341846 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.341950 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 16:32:13.342301 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.335976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45111 (local address 127.5.232.131:58753)
0929 16:32:13.336247 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:13.336271 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:13.336405 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:13.336969 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:13.336974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:13.337006 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:13.337294 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.337303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.338331 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.338343 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.340287 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.340299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.341434 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.341441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.341673 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.341707 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:32:13.341810 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:32:13.341951 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":47,"spinlock_wait_cycles":24960}
I20260929 16:32:13.343328 20725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36395d44e4e480eab4e5df24bb6c348" candidate_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75486dabdb8b4ad188350506644cd676" is_pre_election: true
I20260929 16:32:13.343750 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.336382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:40345 (local address 127.5.232.129:40685)
0929 16:32:13.336697 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.336704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.337105 (+ 401us) server_negotiation.cc:407] Connection header received
0929 16:32:13.337247 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.337250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.337306 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.337428 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:13.338893 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.339928 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.341669 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.342823 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.343011 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.343083 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.343360 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.343488 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260929 16:32:13.343791 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.336020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:40685 (local address 127.5.232.131:40345)
0929 16:32:13.336975 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:13.336992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:13.337119 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:13.338244 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:13.338248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:13.338269 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:13.338691 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.338701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.340072 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.340076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.341506 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:13.341518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.343024 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.343030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:13.343203 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.343226 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:32:13.343333 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:32:13.343465 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":558,"threads_started":1}
I20260929 16:32:13.343897 20725 raft_consensus.cc:2454] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4cdfd7888934fa0ba8f2a8104cd28f4 in term 0.
I20260929 16:32:13.344619 20789 leader_election.cc:302] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75486dabdb8b4ad188350506644cd676, b4cdfd7888934fa0ba8f2a8104cd28f4; no voters:
I20260929 16:32:13.345031 20921 raft_consensus.cc:2781] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:32:13.344871 20593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36395d44e4e480eab4e5df24bb6c348" candidate_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944912a5f3834f6db44b26e9ccb334f4" is_pre_election: true
I20260929 16:32:13.345194 20921 raft_consensus.cc:482] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:32:13.345309 20921 raft_consensus.cc:3037] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:13.345479 20593 raft_consensus.cc:2454] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4cdfd7888934fa0ba8f2a8104cd28f4 in term 0.
I20260929 16:32:13.347759 20921 raft_consensus.cc:504] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:13.348326 20921 leader_election.cc:288] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [CANDIDATE]: Term 1 election: Requested vote from peers 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685), 75486dabdb8b4ad188350506644cd676 (127.5.232.130:45111)
I20260929 16:32:13.348860 20593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36395d44e4e480eab4e5df24bb6c348" candidate_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944912a5f3834f6db44b26e9ccb334f4"
I20260929 16:32:13.348862 20725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36395d44e4e480eab4e5df24bb6c348" candidate_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75486dabdb8b4ad188350506644cd676"
I20260929 16:32:13.349078 20725 raft_consensus.cc:3037] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:13.349094 20593 raft_consensus.cc:3037] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:13.352432 20725 raft_consensus.cc:2454] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4cdfd7888934fa0ba8f2a8104cd28f4 in term 1.
I20260929 16:32:13.352432 20593 raft_consensus.cc:2454] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4cdfd7888934fa0ba8f2a8104cd28f4 in term 1.
I20260929 16:32:13.352929 20790 leader_election.cc:302] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944912a5f3834f6db44b26e9ccb334f4, b4cdfd7888934fa0ba8f2a8104cd28f4; no voters:
I20260929 16:32:13.353262 20921 raft_consensus.cc:2781] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:32:13.353807 20921 raft_consensus.cc:686] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 LEADER]: Becoming Leader. State: Replica: b4cdfd7888934fa0ba8f2a8104cd28f4, State: Running, Role: LEADER
I20260929 16:32:13.354383 20921 consensus_queue.cc:237] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } }
I20260929 16:32:13.359760 20453 catalog_manager.cc:5495] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4cdfd7888934fa0ba8f2a8104cd28f4 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 46153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 } health_report { overall_health: UNKNOWN } } }
W20260929 16:32:13.386379 20902 tablet.cc:2369] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:13.397997 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.392520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.5.232.131:46153)
0929 16:32:13.392708 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.392713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.392816 (+ 103us) server_negotiation.cc:407] Connection header received
0929 16:32:13.392966 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.392970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.393046 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.393178 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:32:13.393577 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.394325 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.395853 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.396265 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.396364 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.396822 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:32:13.397333 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:32:13.397538 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.397620 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.397696 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:32:13.402532 20771 tablet.cc:2369] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:13.406265 20725 raft_consensus.cc:1261] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [term 1 FOLLOWER]: Refusing update from remote peer b4cdfd7888934fa0ba8f2a8104cd28f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:13.406266 20593 raft_consensus.cc:1261] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [term 1 FOLLOWER]: Refusing update from remote peer b4cdfd7888934fa0ba8f2a8104cd28f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:13.407261 20927 consensus_queue.cc:1035] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75486dabdb8b4ad188350506644cd676" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:13.407670 20921 consensus_queue.cc:1035] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "944912a5f3834f6db44b26e9ccb334f4" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 40685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:13.439842 20453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:13.440193 20453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:13.443958 20453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Sending DeleteTablet for 3 replicas of tablet d36395d44e4e480eab4e5df24bb6c348
I20260929 16:32:13.445245 20573 tablet_service.cc:1514] Processing DeleteTablet for tablet d36395d44e4e480eab4e5df24bb6c348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:13 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:44426
I20260929 16:32:13.445144 20836 tablet_service.cc:1514] Processing DeleteTablet for tablet d36395d44e4e480eab4e5df24bb6c348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:13 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:49646
I20260929 16:32:13.445978 20705 tablet_service.cc:1514] Processing DeleteTablet for tablet d36395d44e4e480eab4e5df24bb6c348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:13 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:36080
I20260929 16:32:13.446894 20942 tablet_replica.cc:330] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: stopping tablet replica
I20260929 16:32:13.447480 20942 raft_consensus.cc:2229] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:32:13.448127 20942 raft_consensus.cc:2258] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:13.448347 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 20490
I20260929 16:32:13.456342 20940 tablet_replica.cc:330] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: stopping tablet replica
I20260929 16:32:13.456701 20941 tablet_replica.cc:330] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: stopping tablet replica
I20260929 16:32:13.456832 20940 raft_consensus.cc:2229] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:32:13.457136 20941 raft_consensus.cc:2229] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:32:13.457422 20940 raft_consensus.cc:2258] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:13.457916 20941 raft_consensus.cc:2258] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:13.459767 20942 ts_tablet_manager.cc:1905] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:13.459970 20940 ts_tablet_manager.cc:1905] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:13.463290 20941 ts_tablet_manager.cc:1905] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:13.467144 20942 ts_tablet_manager.cc:1918] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:13.467347 20942 log.cc:1196] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/d36395d44e4e480eab4e5df24bb6c348
I20260929 16:32:13.467453 20940 ts_tablet_manager.cc:1918] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:13.467662 20940 log.cc:1196] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/d36395d44e4e480eab4e5df24bb6c348
I20260929 16:32:13.467983 20942 ts_tablet_manager.cc:1939] T d36395d44e4e480eab4e5df24bb6c348 P 75486dabdb8b4ad188350506644cd676: Deleting consensus metadata
I20260929 16:32:13.468225 20940 ts_tablet_manager.cc:1939] T d36395d44e4e480eab4e5df24bb6c348 P 944912a5f3834f6db44b26e9ccb334f4: Deleting consensus metadata
I20260929 16:32:13.470176 20438 catalog_manager.cc:4841] TS 944912a5f3834f6db44b26e9ccb334f4 (127.5.232.129:40685): tablet d36395d44e4e480eab4e5df24bb6c348 (table test-table [id=bcc20aa37e3f43bd8e4c0b3bbdc4ad6d]) successfully deleted
I20260929 16:32:13.470526 20437 catalog_manager.cc:4841] TS 75486dabdb8b4ad188350506644cd676 (127.5.232.130:45111): tablet d36395d44e4e480eab4e5df24bb6c348 (table test-table [id=bcc20aa37e3f43bd8e4c0b3bbdc4ad6d]) successfully deleted
I20260929 16:32:13.476152 20941 ts_tablet_manager.cc:1918] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:13.476367 20941 log.cc:1196] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/d36395d44e4e480eab4e5df24bb6c348
I20260929 16:32:13.477059 20941 ts_tablet_manager.cc:1939] T d36395d44e4e480eab4e5df24bb6c348 P b4cdfd7888934fa0ba8f2a8104cd28f4: Deleting consensus metadata
I20260929 16:32:13.478034 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.449813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.5.232.129:40685)
0929 16:32:13.450068 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.450077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.450102 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:32:13.450192 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.450201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.450308 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.450442 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:13.453213 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.454857 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.458116 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.458886 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.459020 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.469310 (+ 10290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:13.469328 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:13.469346 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:13.469471 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:13.474089 (+ 4618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.475165 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.475176 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.475180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.475351 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.475840 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.475848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.475854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.476437 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:13.476796 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.477163 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.477462 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 16:32:13.478835 20633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:13.479094 20438 catalog_manager.cc:4841] TS b4cdfd7888934fa0ba8f2a8104cd28f4 (127.5.232.131:46153): tablet d36395d44e4e480eab4e5df24bb6c348 (table test-table [id=bcc20aa37e3f43bd8e4c0b3bbdc4ad6d]) successfully deleted
I20260929 16:32:13.570904 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 20490
I20260929 16:32:13.589785 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 20642
I20260929 16:32:13.604269 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.590609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.5.232.130:45111)
0929 16:32:13.590798 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.590806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.590830 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:32:13.591037 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.591043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.591122 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.591227 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:13.592670 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.593460 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.594894 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.595294 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.595431 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.598131 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:13.598147 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:13.598163 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:13.598208 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:13.600973 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.602063 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.602073 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.602077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.602215 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.602619 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.602624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.602628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.603025 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:13.603352 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.603690 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.603916 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:32:13.605008 20765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:13.705704 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 20642
I20260929 16:32:13.725200 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 20773
I20260929 16:32:13.741773 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:13.726352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.5.232.131:46153)
0929 16:32:13.726784 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:13.726792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:13.726818 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:32:13.726894 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:13.726900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:13.726997 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:13.727130 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:13.729138 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.730172 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.731693 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:13.732152 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:13.732308 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:13.735175 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:13.735191 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:13.735211 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:13.735256 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:13.738558 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.739347 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.739360 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.739368 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.739539 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:13.740009 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:13.740014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:13.740020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:13.740425 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:13.740779 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:13.741087 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 16:32:13.741347 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260929 16:32:13.742785 20896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:13.869419 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 20773
I20260929 16:32:13.890152 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 20420
I20260929 16:32:13.891319 20483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:14.007586 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 20420
I20260929 16:32:14.032428 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42935
--webserver_interface=127.5.232.190
--webserver_port=35541
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:14.376670 20947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:14.377070 20947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:14.377257 20947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:14.384956 20947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:14.385133 20947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:14.385248 20947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:14.385346 20947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:14.385450 20947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:14.396152 20947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=35541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:14.397821 20947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:14.399179 20947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:14.412536 20952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:14.412488 20955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:14.412443 20953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:14.413223 20947 server_base.cc:1034] running on GCE node
I20260929 16:32:14.414155 20947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:14.415714 20947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:14.416944 20947 hybrid_clock.cc:648] HybridClock initialized: now 1790699534416871 us; error 58 us; skew 500 ppm
I20260929 16:32:14.417356 20947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:14 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699534, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:14.420733 20947 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:32:14.422456 20947 webserver.cc:460] Webserver started at http://127.5.232.190:35541/ using document root <none> and password file <none>
I20260929 16:32:14.423218 20947 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:14.423360 20947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:14.429324 20947 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:32:14.432626 20962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:14.434213 20947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:32:14.434408 20947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "e3342ffbcf0d421eb102d6a3cee2d929"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "4c52e4caf8bf994cd9e492e6e4a52da8"
server_key_iv: "8025bebd3de18f9c7e231f7a39547649"
server_key_version: "encryptionkey@0"
I20260929 16:32:14.435462 20947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:14.492190 20947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:14.496677 20947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:14.497089 20947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:14.516822 20947 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:42935
I20260929 16:32:14.516840 21013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:42935 every 8 connection(s)
I20260929 16:32:14.518466 20947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:32:14.525636 21014 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:32:14.527060 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 20947
I20260929 16:32:14.528080 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:40685
--local_ip_for_outbound_sockets=127.5.232.129
--tserver_master_addrs=127.5.232.190:42935
--webserver_port=41587
--webserver_interface=127.5.232.129
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:32:14.530431 21014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Bootstrap starting.
I20260929 16:32:14.553297 21014 log.cc:824] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:14.570641 21014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:32:14.571223 21014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929: Bootstrap complete.
I20260929 16:32:14.580725 21014 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:14.581792 21014 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3342ffbcf0d421eb102d6a3cee2d929, State: Initialized, Role: FOLLOWER
I20260929 16:32:14.582433 21014 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:14.582710 21014 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:32:14.582849 21014 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:32:14.583032 21014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:32:14.587164 21014 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:14.587718 21014 leader_election.cc:302] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3342ffbcf0d421eb102d6a3cee2d929; no voters:
I20260929 16:32:14.588405 21014 leader_election.cc:288] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:32:14.588560 21018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:32:14.589797 21018 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [term 3 LEADER]: Becoming Leader. State: Replica: e3342ffbcf0d421eb102d6a3cee2d929, State: Running, Role: LEADER
I20260929 16:32:14.590391 21018 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } }
I20260929 16:32:14.590843 21014 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:32:14.593088 21020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3342ffbcf0d421eb102d6a3cee2d929. Latest consensus state: current_term: 3 leader_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } } }
I20260929 16:32:14.593178 21019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3342ffbcf0d421eb102d6a3cee2d929" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42935 } } }
I20260929 16:32:14.593854 21020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:14.593904 21019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3342ffbcf0d421eb102d6a3cee2d929 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:14.606986 21025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:32:14.619011 21025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:32:14.621136 21025 catalog_manager.cc:1259] Loaded cluster ID: 2bcd1b30b116434a9f0b660fef3f7480
I20260929 16:32:14.621280 21025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 16:32:14.624370 21025 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd9914b69ad google::LogMessage::Fail() at ??:0
@ 0x7fd9914b988a google::LogMessage::SendToLog() at ??:0
@ 0x7fd9914b64de google::LogMessage::Flush() at ??:0
@ 0x7fd9914b7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd9a6ee9512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd9a6ee6976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd9a6f6e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd9a6f6e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
I20260929 16:32:14.677889 21037 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fd994102345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd99410591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd994105799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd9940e0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd9973ba6db start_thread at ??:0
@ 0x7fd99035061f clone at ??:0
W20260929 16:32:14.939558 21016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:14.939911 21016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:14.940083 21016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:14.947830 21016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:14.948006 21016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:14.948185 21016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:32:14.958711 21016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:40685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=41587
--tserver_master_addrs=127.5.232.190:42935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:14.960132 21016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:14.961656 21016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:14.975582 21055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:14.975544 21056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:14.975466 21058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:14.976626 21016 server_base.cc:1034] running on GCE node
I20260929 16:32:14.977300 21016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:14.978394 21016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:14.979604 21016 hybrid_clock.cc:648] HybridClock initialized: now 1790699534979559 us; error 39 us; skew 500 ppm
I20260929 16:32:14.979965 21016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:14 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699534, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:14.983383 21016 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:32:14.985184 21016 webserver.cc:460] Webserver started at http://127.5.232.129:41587/ using document root <none> and password file <none>
I20260929 16:32:14.986019 21016 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:14.986176 21016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:14.992053 21016 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:32:14.995148 21065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:14.996549 21016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:32:14.996737 21016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "944912a5f3834f6db44b26e9ccb334f4"
format_stamp: "Formatted at 2026-09-29 16:32:08 on dist-test-slave-dssx"
server_key: "fe572d280e2dd3f667986078fbbf8259"
server_key_iv: "d09d10386db0882bf21703e041f4ef2a"
server_key_version: "encryptionkey@0"
I20260929 16:32:14.997869 21016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:15.013093 21016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:15.017369 21016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:15.017735 21016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:15.018766 21016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:15.020141 21016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:15.020285 21016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:15.020416 21016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:15.020505 21016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:15.055519 21016 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:40685
I20260929 16:32:15.055569 21177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:40685 every 8 connection(s)
I20260929 16:32:15.057058 21016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:32:15.062292 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:15.060300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:38403)
0929 16:32:15.061059 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:15.061296 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":297,"threads_started":1}
W20260929 16:32:15.063042 21178 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:42935 (0 consecutive failures): Network error: Failed to ping master at 127.5.232.190:42935: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
I20260929 16:32:15.063931 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:15.063598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:44501)
0929 16:32:15.063748 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:15.063822 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:32:15.064759 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:15.064454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:56515)
0929 16:32:15.064571 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:15.064645 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:32:15.065014 21178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:32:15.065855 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 21016
I20260929 16:32:15.066452 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:45111
--local_ip_for_outbound_sockets=127.5.232.130
--tserver_master_addrs=127.5.232.190:42935
--webserver_port=42403
--webserver_interface=127.5.232.130
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:15.404337 21181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:15.404639 21181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:15.404803 21181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:15.412321 21181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:15.412456 21181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:15.412679 21181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:32:15.423106 21181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:45111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=42403
--tserver_master_addrs=127.5.232.190:42935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:15.424510 21181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:15.425885 21181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:15.439265 21186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:15.439304 21189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:15.439333 21187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:15.440301 21181 server_base.cc:1034] running on GCE node
I20260929 16:32:15.440953 21181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:15.442014 21181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:15.443271 21181 hybrid_clock.cc:648] HybridClock initialized: now 1790699535443214 us; error 49 us; skew 500 ppm
I20260929 16:32:15.443612 21181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:15 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699535, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:15.446859 21181 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:32:15.448513 21181 webserver.cc:460] Webserver started at http://127.5.232.130:42403/ using document root <none> and password file <none>
I20260929 16:32:15.449301 21181 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:15.449481 21181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:15.455065 21181 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:32:15.458379 21196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:15.459829 21181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:32:15.460042 21181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "75486dabdb8b4ad188350506644cd676"
format_stamp: "Formatted at 2026-09-29 16:32:09 on dist-test-slave-dssx"
server_key: "a0b80b3d61393410547cf9265e86aaa6"
server_key_iv: "6a46d744e67c464f2248e3b489b19a3e"
server_key_version: "encryptionkey@0"
I20260929 16:32:15.461077 21181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:15.487903 21181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:15.492209 21181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:15.492522 21181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:15.493739 21181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:15.495127 21181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:15.495270 21181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:15.495410 21181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:15.495507 21181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:15.529965 21181 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45111
I20260929 16:32:15.530057 21308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45111 every 8 connection(s)
I20260929 16:32:15.531495 21181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:32:15.536537 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:15.534697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:38829)
0929 16:32:15.535425 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:15.535668 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":279,"threads_started":1}
W20260929 16:32:15.537350 21309 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:42935 (0 consecutive failures): Network error: Failed to ping master at 127.5.232.190:42935: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
I20260929 16:32:15.538234 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:15.537918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:49963)
0929 16:32:15.538057 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:15.538126 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 16:32:15.539101 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:15.538805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:36195)
0929 16:32:15.538925 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:15.538996 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:32:15.539388 21309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:32:15.541546 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 21181
I20260929 16:32:15.542172 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:46153
--local_ip_for_outbound_sockets=127.5.232.131
--tserver_master_addrs=127.5.232.190:42935
--webserver_port=44621
--webserver_interface=127.5.232.131
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:15.877830 21312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:15.878134 21312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:15.878300 21312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:15.885630 21312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:15.885744 21312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:15.885938 21312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:32:15.895900 21312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:46153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=44621
--tserver_master_addrs=127.5.232.190:42935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:15.897245 21312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:15.898602 21312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:15.912144 21318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:15.912256 21317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:15.912226 21320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:15.912925 21312 server_base.cc:1034] running on GCE node
I20260929 16:32:15.913662 21312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:15.914657 21312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:15.915901 21312 hybrid_clock.cc:648] HybridClock initialized: now 1790699535915860 us; error 57 us; skew 500 ppm
I20260929 16:32:15.916244 21312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:15 dist-test-slave-dssx krb5kdc[19866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699535, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:15.919391 21312 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:32:15.920873 21312 webserver.cc:460] Webserver started at http://127.5.232.131:44621/ using document root <none> and password file <none>
I20260929 16:32:15.921690 21312 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:15.921842 21312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:15.927260 21312 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:32:15.930219 21327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:15.931879 21312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:32:15.932073 21312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "b4cdfd7888934fa0ba8f2a8104cd28f4"
format_stamp: "Formatted at 2026-09-29 16:32:10 on dist-test-slave-dssx"
server_key: "e12776b9d86695969fa02945942d2dd8"
server_key_iv: "054765d50f8fd731acab7f8c091c3a6a"
server_key_version: "encryptionkey@0"
I20260929 16:32:15.933095 21312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:15.955729 21312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:15.960418 21312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:15.960760 21312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:15.961969 21312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:15.963390 21312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:15.963517 21312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:15.963652 21312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:15.963750 21312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:15.996327 21312 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:46153
I20260929 16:32:15.996382 21439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:46153 every 8 connection(s)
I20260929 16:32:15.997989 21312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:32:16.003300 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:16.001090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:33913)
0929 16:32:16.002078 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:16.002337 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":350,"threads_started":1}
W20260929 16:32:16.003983 21440 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:42935 (0 consecutive failures): Network error: Failed to ping master at 127.5.232.190:42935: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
I20260929 16:32:16.004839 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:16.004518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:33387)
0929 16:32:16.004671 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:16.004739 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:32:16.005311 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 21312
I20260929 16:32:16.005822 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:16.005511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:49221)
0929 16:32:16.005644 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:16.005714 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:32:16.006079 21440 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:32:16.066543 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:16.065664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:44029)
0929 16:32:16.066205 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:16.066317 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":299,"threads_started":1}
I20260929 16:32:16.540796 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:16.539952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:47793)
0929 16:32:16.540461 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:16.540558 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":257,"threads_started":1}
I20260929 16:32:17.007310 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:17.006548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:51217)
0929 16:32:17.006998 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:17.007084 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":227,"threads_started":1}
I20260929 16:32:17.068284 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:17.067522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:47663)
0929 16:32:17.068023 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:17.068109 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
I20260929 16:32:17.542516 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:17.541820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:35185)
0929 16:32:17.542271 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:17.542352 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260929 16:32:18.008984 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:18.008204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:33965)
0929 16:32:18.008682 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:18.008776 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":232,"threads_started":1}
I20260929 16:32:18.070039 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:18.069201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:43673)
0929 16:32:18.069778 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:18.069867 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":306,"threads_started":1}
I20260929 16:32:18.544162 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:18.543442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:57219)
0929 16:32:18.543896 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:18.543982 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260929 16:32:19.010753 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:19.010055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:39037)
0929 16:32:19.010497 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:19.010579 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":237,"threads_started":1}
I20260929 16:32:19.071753 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:19.071051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:39777)
0929 16:32:19.071483 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:19.071564 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260929 16:32:19.545941 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:19.545116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:56993)
0929 16:32:19.545672 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:19.545761 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":308,"threads_started":1}
I20260929 16:32:20.012427 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:20.011691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:33715)
0929 16:32:20.012145 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:20.012228 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260929 16:32:20.073622 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:20.072726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:51033)
0929 16:32:20.073204 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:20.073289 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":241,"threads_started":1}
I20260929 16:32:20.547657 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:20.546883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:51941)
0929 16:32:20.547393 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:20.547471 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":285,"threads_started":1}
I20260929 16:32:21.014175 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:21.013465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:56099)
0929 16:32:21.013923 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:21.014011 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":217,"threads_started":1}
I20260929 16:32:21.075430 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:21.074546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:36893)
0929 16:32:21.075151 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:21.075272 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":347,"threads_started":1}
I20260929 16:32:21.549592 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:21.548681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:36529)
0929 16:32:21.549262 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:21.549354 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":336,"threads_started":1}
I20260929 16:32:22.015842 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:22.015125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:39741)
0929 16:32:22.015574 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:22.015670 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260929 16:32:22.077257 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:22.076436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:49065)
0929 16:32:22.076948 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:22.077040 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":220,"threads_started":1}
I20260929 16:32:22.551544 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:22.550646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:59795)
0929 16:32:22.551249 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:22.551338 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":270,"threads_started":1}
I20260929 16:32:23.017752 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:23.016889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:46281)
0929 16:32:23.017441 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:23.017532 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":225,"threads_started":1}
I20260929 16:32:23.079154 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:23.078237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:55943)
0929 16:32:23.078862 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:23.078978 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":338,"threads_started":1}
I20260929 16:32:23.553678 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:23.552612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:48923)
0929 16:32:23.553200 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:23.553419 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":343,"threads_started":1}
I20260929 16:32:24.019382 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:24.018656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:38523)
0929 16:32:24.019111 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:24.019215 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":230,"threads_started":1}
I20260929 16:32:24.080948 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:24.080178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:34185)
0929 16:32:24.080680 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:24.080769 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":254,"threads_started":1}
I20260929 16:32:24.555552 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:24.554770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:52201)
0929 16:32:24.555279 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:24.555367 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":244,"threads_started":1}
I20260929 16:32:25.021134 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:25.020351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:38333)
0929 16:32:25.020866 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:25.020959 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":229,"threads_started":1}
I20260929 16:32:25.082947 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:25.082081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:53751)
0929 16:32:25.082694 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:25.082777 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":331,"threads_started":1}
I20260929 16:32:25.557545 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:25.556664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:44279)
0929 16:32:25.557223 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:25.557305 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":334,"threads_started":1}
I20260929 16:32:26.022945 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:26.022164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:34313)
0929 16:32:26.022688 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:26.022777 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":289,"threads_started":1}
I20260929 16:32:26.084769 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:26.084007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:51649)
0929 16:32:26.084483 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:26.084587 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":226,"threads_started":1}
I20260929 16:32:26.559324 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:26.558564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:32889)
0929 16:32:26.559027 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:26.559131 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":232,"threads_started":1}
I20260929 16:32:27.024770 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:27.023959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:35851)
0929 16:32:27.024484 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:27.024580 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":288,"threads_started":1}
I20260929 16:32:27.086719 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:27.086003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:59253)
0929 16:32:27.086457 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:27.086554 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":225,"threads_started":1}
I20260929 16:32:27.561090 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:27.560409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:47297)
0929 16:32:27.560847 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:27.560934 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":220,"threads_started":1}
I20260929 16:32:28.026386 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:28.025660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:57515)
0929 16:32:28.026122 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:28.026208 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260929 16:32:28.088712 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:28.087912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:47531)
0929 16:32:28.088412 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:28.088518 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":235,"threads_started":1}
I20260929 16:32:28.562997 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:28.562181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:57903)
0929 16:32:28.562698 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:28.562802 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":257,"threads_started":1}
I20260929 16:32:29.028134 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:29.027349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:55245)
0929 16:32:29.027834 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:29.027944 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":227,"threads_started":1}
I20260929 16:32:29.090492 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:29.089761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:59643)
0929 16:32:29.090205 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:29.090300 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260929 16:32:29.564658 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:29.563959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:35779)
0929 16:32:29.564392 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:29.564475 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260929 16:32:30.030046 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:30.029230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:41319)
0929 16:32:30.029761 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:30.029852 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":279,"threads_started":1}
I20260929 16:32:30.092417 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:30.091649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.129:58923)
0929 16:32:30.092134 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:30.092232 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260929 16:32:30.566745 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:30.565973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.130:55231)
0929 16:32:30.566434 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:30.566559 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":237,"threads_started":1}
I20260929 16:32:31.016774 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 21016
I20260929 16:32:31.031872 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:31.031146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42935 (local address 127.5.232.131:37081)
0929 16:32:31.031578 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:31.031690 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":205,"threads_started":1}
I20260929 16:32:31.038187 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:31.018064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.5.232.129:40685)
0929 16:32:31.018874 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:31.018896 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:31.018947 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:32:31.021255 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:31.021266 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:31.021823 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:31.022140 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:31.023667 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:31.024965 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:31.026449 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:31.027175 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:31.027325 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:31.030509 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:31.030518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:31.030536 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:31.030581 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:31.034713 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:31.035465 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:31.035473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:31.035480 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:31.035665 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:31.035995 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:31.035998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:31.036000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:31.036547 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:31.036917 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:31.037437 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 16:32:31.037906 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":507,"threads_started":1}
I20260929 16:32:31.039697 21173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:31.131366 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 21016
I20260929 16:32:31.148967 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 21181
I20260929 16:32:31.168282 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:31.150014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.5.232.130:45111)
0929 16:32:31.150578 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:31.150592 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:31.150634 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:32:31.152169 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:31.152182 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:31.152670 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:31.152996 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:31.154550 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:31.155849 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:31.157441 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:31.158239 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:31.158406 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:31.161229 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:31.161238 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:31.161257 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:31.161305 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:31.164500 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:31.165572 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:31.165580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:31.165587 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:31.165856 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:31.166256 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:31.166259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:31.166261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:31.166853 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:31.167104 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:31.167489 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 16:32:31.167977 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":264,"threads_started":1}
I20260929 16:32:31.169836 21304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:31.254725 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 21181
I20260929 16:32:31.273797 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 21312
I20260929 16:32:31.292021 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:31.274869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.5.232.131:46153)
0929 16:32:31.275620 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:31.275633 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:31.275673 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:32:31.277278 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:31.277289 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:31.277820 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:31.278118 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:31.279530 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:31.280769 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:31.282168 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:31.282847 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:31.282986 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:31.285640 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:31.285648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:31.285666 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:31.285715 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:31.288797 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:31.289620 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:31.289627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:31.289634 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:31.289810 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:31.290148 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:31.290152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:31.290154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:31.290722 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:31.290957 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:31.291342 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 16:32:31.291705 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":410,"threads_started":1}
I20260929 16:32:31.293654 21435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:31.378194 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 21312
2026-09-29T16:32:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24763 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:32:31 dist-test-slave-dssx krb5kdc[21498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:32:31 dist-test-slave-dssx krb5kdc[21498](info): set up 2 sockets
Sep 29 16:32:31 dist-test-slave-dssx krb5kdc[21498](info): commencing operation
krb5kdc: starting...
W20260929 16:32:32.490118 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:32:32 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:32:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:32:32.734905 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42825
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:33.069413 21511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:33.069721 21511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:33.069814 21511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:33.078318 21511 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:32:33.078423 21511 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:32:33.078464 21511 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:32:33.078521 21511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:33.078575 21511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:33.078642 21511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:33.078686 21511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:33.078739 21511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:33.088999 21511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:33.090372 21511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:33.091645 21511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:32:33.104343 21511 server_base.cc:1034] running on GCE node
W20260929 16:32:33.104300 21519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:33.104221 21517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:33.104261 21516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:33.105727 21511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:33.107225 21511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:33.108445 21511 hybrid_clock.cc:648] HybridClock initialized: now 1790699553108387 us; error 40 us; skew 500 ppm
I20260929 16:32:33.108795 21511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:33 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:33.115100 21511 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:32:33.116770 21511 webserver.cc:460] Webserver started at http://127.5.232.190:46425/ using document root <none> and password file <none>
I20260929 16:32:33.117565 21511 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:33.117681 21511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:33.118027 21511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:33.121111 21511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "a386273cf011d27e015bb7838c25d23b"
server_key_iv: "720ff0ea643584211674f976f68f2b19"
server_key_version: "encryptionkey@0"
I20260929 16:32:33.122011 21511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "a386273cf011d27e015bb7838c25d23b"
server_key_iv: "720ff0ea643584211674f976f68f2b19"
server_key_version: "encryptionkey@0"
I20260929 16:32:33.127470 21511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:32:33.131143 21526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:33.132637 21511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:32:33.132834 21511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "a386273cf011d27e015bb7838c25d23b"
server_key_iv: "720ff0ea643584211674f976f68f2b19"
server_key_version: "encryptionkey@0"
I20260929 16:32:33.133064 21511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:33.144832 21511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:33.149247 21511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:33.149572 21511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:33.168771 21511 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:42825
I20260929 16:32:33.168789 21577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:42825 every 8 connection(s)
I20260929 16:32:33.170233 21511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:32:33.173982 21578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:33.178143 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 21511
I20260929 16:32:33.178396 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:32:33.179082 6050 external_mini_cluster.cc:1428] Setting key 89ac0d16da3bf8542b719da9a60ff811
I20260929 16:32:33.185446 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Bootstrap starting.
I20260929 16:32:33.189818 21578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:33.191053 21578 log.cc:824] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:33.194144 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: No bootstrap required, opened a new log
Sep 29 16:32:33 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:33.203029 21578 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:32:33.203473 21578 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:33.203588 21578 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7eaf0bd0d94a6daf8b8041e7f10ed8, State: Initialized, Role: FOLLOWER
I20260929 16:32:33.204293 21578 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:32:33.204623 21578 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:32:33.204789 21578 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:32:33.204975 21578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:33.207971 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:33.181356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.5.232.190:42825)
0929 16:32:33.182433 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:33.182450 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:33.186088 (+ 3638us) server_negotiation.cc:407] Connection header received
0929 16:32:33.188723 (+ 2635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:33.188760 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:33.189333 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:33.189863 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:33.191353 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:33.192706 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:33.194139 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:33.194884 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:33.195109 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:33.199493 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:33.199504 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:33.199521 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:33.199566 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:33.202858 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:33.203708 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:33.203717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:33.203726 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:33.203962 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:33.204379 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:33.204384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:33.204387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:33.205218 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:33.205605 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:33.205989 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 16:32:33.206512 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":460,"threads_started":1}
I20260929 16:32:33.208446 21578 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:32:33.209000 21578 leader_election.cc:302] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf7eaf0bd0d94a6daf8b8041e7f10ed8; no voters:
I20260929 16:32:33.209645 21578 leader_election.cc:288] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:32:33.209808 21583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:32:33.210918 21583 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 1 LEADER]: Becoming Leader. State: Replica: bf7eaf0bd0d94a6daf8b8041e7f10ed8, State: Running, Role: LEADER
I20260929 16:32:33.211563 21583 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:32:33.212006 21578 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:32:33.214437 21585 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf7eaf0bd0d94a6daf8b8041e7f10ed8. Latest consensus state: current_term: 1 leader_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } } }
I20260929 16:32:33.214841 21585 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:33.215478 21584 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } } }
I20260929 16:32:33.215768 21584 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:33.215768 21592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:32:33.220870 21592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:32:33.230031 21592 catalog_manager.cc:1347] Generated new cluster ID: b26230192ffb438396ce0d22c645caf5
I20260929 16:32:33.230165 21592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:32:33.259385 21592 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:32:33.261411 21592 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:32:33.280659 21592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Generated new TSK 0
I20260929 16:32:33.281661 21592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:32:33.342610 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42825
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:33.692441 21604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:33.692734 21604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:33.692903 21604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:33.700855 21604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:33.700984 21604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:33.701198 21604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:32:33.711609 21604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:33.712978 21604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:33.714457 21604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:33.729952 21609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:33.729982 21612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:33.729903 21610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:33.730592 21604 server_base.cc:1034] running on GCE node
I20260929 16:32:33.731310 21604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:33.732278 21604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:33.733495 21604 hybrid_clock.cc:648] HybridClock initialized: now 1790699553733431 us; error 51 us; skew 500 ppm
I20260929 16:32:33.733831 21604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:33 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:33.737196 21604 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:32:33.738765 21604 webserver.cc:460] Webserver started at http://127.5.232.129:34527/ using document root <none> and password file <none>
I20260929 16:32:33.739440 21604 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:33.739543 21604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:33.739809 21604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:33.742826 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "88a0a340f85240b3a5581a550267ab0f"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "90cb1422341932745693163b61e0f2f2"
server_key_iv: "aed754f53b1f5659ec3e733d305cdb83"
server_key_version: "encryptionkey@0"
I20260929 16:32:33.743579 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "88a0a340f85240b3a5581a550267ab0f"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "90cb1422341932745693163b61e0f2f2"
server_key_iv: "aed754f53b1f5659ec3e733d305cdb83"
server_key_version: "encryptionkey@0"
I20260929 16:32:33.749065 21604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:32:33.752823 21619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:33.754231 21604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:32:33.754369 21604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "88a0a340f85240b3a5581a550267ab0f"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "90cb1422341932745693163b61e0f2f2"
server_key_iv: "aed754f53b1f5659ec3e733d305cdb83"
server_key_version: "encryptionkey@0"
I20260929 16:32:33.754561 21604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:33.774808 21604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:33.778855 21604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:33.779112 21604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:33.780021 21604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:33.781337 21604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:33.781488 21604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:33.781591 21604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:33.781638 21604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:33.814558 21604 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:41959
I20260929 16:32:33.814597 21731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:41959 every 8 connection(s)
I20260929 16:32:33.815982 21604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:32:33.817332 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 21604
I20260929 16:32:33.817582 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:32:33.818363 6050 external_mini_cluster.cc:1428] Setting key bae13e081e33185e7cb93c114bcad8d8
Sep 29 16:32:33 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699553, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:33.838374 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:33.818748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:52727 (local address 127.5.232.190:42825)
0929 16:32:33.819259 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:33.819266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:33.820379 (+ 1113us) server_negotiation.cc:407] Connection header received
0929 16:32:33.822314 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:33.822320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:33.822409 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:33.822524 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:33.824046 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:33.824759 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:33.826397 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:33.826808 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:33.826944 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:33.831027 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:33.831045 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:33.831053 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:33.831106 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:33.834664 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:33.835458 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:33.835467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:33.835473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:33.835627 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:33.836160 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:33.836168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:33.836173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:33.836573 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:33.836750 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:33.837558 (+ 808us) server_negotiation.cc:299] Negotiation successful
0929 16:32:33.837841 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":221,"threads_started":1}
I20260929 16:32:33.839171 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:33.819365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:52727)
0929 16:32:33.820215 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:33.820286 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:33.822047 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:33.822766 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:33.822775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:33.823301 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:33.823868 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:33.823886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:33.824923 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:33.824927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:33.826250 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:33.826263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:33.827049 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:33.827056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:33.827287 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:33.828068 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:33.828102 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:33.830787 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:33.834829 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:33.834837 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:33.834852 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:33.835313 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:33.835777 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:33.835784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:33.835790 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:33.836036 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:33.836698 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:33.836705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:33.837115 (+ 410us) client_negotiation.cc:769] Sending connection context
0929 16:32:33.837356 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 16:32:33.837877 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":288,"threads_started":1}
I20260929 16:32:33.840926 21732 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42825
I20260929 16:32:33.841293 21732 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:33.842268 21732 heartbeater.cc:511] Master 127.5.232.190:42825 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
I20260929 16:32:33.844722 21543 ts_manager.cc:194] Registered new tserver with Master: 88a0a340f85240b3a5581a550267ab0f (127.5.232.129:41959)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
I20260929 16:32:33.846503 21543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:52727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:32:33.872920 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42825
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:34.203636 21738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:34.203997 21738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:34.204172 21738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:34.211575 21738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:34.211750 21738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:34.211921 21738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:32:34.222084 21738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:34.223465 21738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:34.224793 21738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:34.238540 21743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:34.238708 21746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:34.238600 21744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:34.239620 21738 server_base.cc:1034] running on GCE node
I20260929 16:32:34.240249 21738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:34.241364 21738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:34.242643 21738 hybrid_clock.cc:648] HybridClock initialized: now 1790699554242564 us; error 64 us; skew 500 ppm
I20260929 16:32:34.242975 21738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:34 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699554, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:34.246253 21738 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:32:34.247763 21738 webserver.cc:460] Webserver started at http://127.5.232.130:39037/ using document root <none> and password file <none>
I20260929 16:32:34.248512 21738 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:34.248657 21738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:34.248972 21738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:34.252111 21738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "9f75774c887b49b9820de2f25172ae5c"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "3f1b4c94aa21aa47c4e9d9980c07764d"
server_key_iv: "bc0e02312fdc84051c7907aebd7bdd30"
server_key_version: "encryptionkey@0"
I20260929 16:32:34.252929 21738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "9f75774c887b49b9820de2f25172ae5c"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "3f1b4c94aa21aa47c4e9d9980c07764d"
server_key_iv: "bc0e02312fdc84051c7907aebd7bdd30"
server_key_version: "encryptionkey@0"
I20260929 16:32:34.258608 21738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:32:34.262419 21753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:34.263859 21738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:32:34.264070 21738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "9f75774c887b49b9820de2f25172ae5c"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "3f1b4c94aa21aa47c4e9d9980c07764d"
server_key_iv: "bc0e02312fdc84051c7907aebd7bdd30"
server_key_version: "encryptionkey@0"
I20260929 16:32:34.264281 21738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:34.281349 21738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:34.285744 21738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:34.286049 21738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:34.287072 21738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:34.288455 21738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:34.288558 21738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:34.288700 21738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:34.288769 21738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:34.321830 21738 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45039
I20260929 16:32:34.321875 21865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45039 every 8 connection(s)
I20260929 16:32:34.323308 21738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:32:34.326246 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 21738
I20260929 16:32:34.326462 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:32:34.326948 6050 external_mini_cluster.cc:1428] Setting key 153166be800b806deec3f3b2262d5c67
Sep 29 16:32:34 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699554, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:34.347905 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.326028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:46057 (local address 127.5.232.190:42825)
0929 16:32:34.326274 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:34.326282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:34.327638 (+ 1356us) server_negotiation.cc:407] Connection header received
0929 16:32:34.330021 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:34.330025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:34.330096 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:34.330188 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:34.332450 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.333212 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.335905 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.336509 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.336615 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.340572 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:34.340582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:34.340589 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:34.340638 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:34.344349 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:34.345189 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:34.345198 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:34.345203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:34.345414 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:34.345843 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:34.345851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:34.345858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:34.346263 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:34.346432 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:34.347141 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 16:32:34.347418 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:32:34.348620 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.326585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:46057)
0929 16:32:34.327436 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:34.327492 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:34.329656 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:34.330500 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:34.330516 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:34.331360 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:34.332242 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.332272 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.333423 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.333431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.335695 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.335712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.336750 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.336760 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.337126 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.337947 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:34.337987 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:34.340313 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:34.344524 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:34.344531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:34.344547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:34.345029 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:34.345524 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:34.345527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:34.345529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:34.345719 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:34.346403 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:34.346409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:34.346754 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 16:32:34.346994 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 16:32:34.347408 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":339,"threads_started":1}
I20260929 16:32:34.350270 21866 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42825
I20260929 16:32:34.350585 21866 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:34.351334 21866 heartbeater.cc:511] Master 127.5.232.190:42825 requested a full tablet report, sending...
I20260929 16:32:34.352878 21543 ts_manager.cc:194] Registered new tserver with Master: 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039)
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
I20260929 16:32:34.353783 21543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:46057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:32:34.384356 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42825
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:34.718488 21871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:34.718783 21871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:34.718979 21871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:34.726781 21871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:34.726909 21871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:34.727090 21871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:32:34.737427 21871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:34.738893 21871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:34.740234 21871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:34.753937 21877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:34.754009 21876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:34.754009 21879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:34.758950 21871 server_base.cc:1034] running on GCE node
I20260929 16:32:34.759684 21871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:34.760741 21871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:34.761967 21871 hybrid_clock.cc:648] HybridClock initialized: now 1790699554761907 us; error 55 us; skew 500 ppm
I20260929 16:32:34.762316 21871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:34 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699554, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:34.768404 21871 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:32:34.770159 21871 webserver.cc:460] Webserver started at http://127.5.232.131:41165/ using document root <none> and password file <none>
I20260929 16:32:34.770938 21871 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:34.771085 21871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:34.771410 21871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:32:34.774521 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "642f2990e3b84ff7b6c6584fe57df64a"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "788e8cb8ff5274699380a7881682620b"
server_key_iv: "ece20ef5d0418be1d9af8bd3c8451b3c"
server_key_version: "encryptionkey@0"
I20260929 16:32:34.775317 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "642f2990e3b84ff7b6c6584fe57df64a"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "788e8cb8ff5274699380a7881682620b"
server_key_iv: "ece20ef5d0418be1d9af8bd3c8451b3c"
server_key_version: "encryptionkey@0"
I20260929 16:32:34.780821 21871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:32:34.784490 21886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:34.785897 21871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:32:34.786089 21871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "642f2990e3b84ff7b6c6584fe57df64a"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "788e8cb8ff5274699380a7881682620b"
server_key_iv: "ece20ef5d0418be1d9af8bd3c8451b3c"
server_key_version: "encryptionkey@0"
I20260929 16:32:34.786309 21871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:34.815364 21871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:34.820013 21871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:34.820335 21871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:34.821323 21871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:34.822738 21871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:34.822839 21871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:34.822980 21871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:34.823055 21871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:34.849511 21732 heartbeater.cc:503] Master 127.5.232.190:42825 was elected leader, sending a full tablet report...
I20260929 16:32:34.857749 21871 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:38417
I20260929 16:32:34.857784 21998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:38417 every 8 connection(s)
I20260929 16:32:34.859171 21871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:32:34.868641 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 21871
I20260929 16:32:34.868837 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:32:34.869427 6050 external_mini_cluster.cc:1428] Setting key 52a4a692d5785e43b9aa8da23ca84821
Sep 29 16:32:34 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699554, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:34.880540 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.861772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:52963 (local address 127.5.232.190:42825)
0929 16:32:34.862249 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:34.862254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:34.863081 (+ 827us) server_negotiation.cc:407] Connection header received
0929 16:32:34.865252 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:34.865258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:34.865365 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:34.865542 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:34.867139 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.867955 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.870405 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.871031 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.871178 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.874996 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:34.875002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:34.875025 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:34.875063 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:34.877677 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:34.878438 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:34.878442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:34.878444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:34.878570 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:34.878968 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:34.878971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:34.878973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:34.879256 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:34.879375 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:34.880074 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 16:32:34.880254 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":262,"threads_started":1}
I20260929 16:32:34.881672 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.862230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:52963)
0929 16:32:34.862905 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:34.862954 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:34.864958 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:34.865802 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:34.865811 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:34.866347 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:34.866953 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.866971 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.868122 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.868127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.870233 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.870247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.871255 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.871261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.871577 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.872281 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:34.872328 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:34.874757 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:34.877838 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:34.877846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:34.877861 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:34.878328 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:34.878670 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:34.878673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:34.878675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:34.878867 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:34.879368 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:34.879374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:34.879735 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 16:32:34.879972 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 16:32:34.880396 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":260,"threads_started":1}
I20260929 16:32:34.883214 21999 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42825
I20260929 16:32:34.883579 21999 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:34.884550 21999 heartbeater.cc:511] Master 127.5.232.190:42825 requested a full tablet report, sending...
I20260929 16:32:34.886067 21543 ts_manager.cc:194] Registered new tserver with Master: 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417)
I20260929 16:32:34.886921 21543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:52963
I20260929 16:32:34.897508 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:32:34.915237 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.902352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.5.232.190:42825)
0929 16:32:34.902522 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:34.902526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:34.902943 (+ 417us) server_negotiation.cc:407] Connection header received
0929 16:32:34.903133 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:34.903137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:34.903214 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:34.903303 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:34.904787 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.905620 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.906991 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.907395 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.907493 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.909974 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:34.909979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:34.909982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:34.910032 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:34.912636 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:34.913325 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:34.913329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:34.913331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:34.913516 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:34.913860 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:34.913863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:34.913865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:34.914200 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:34.914378 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:34.914688 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 16:32:34.914886 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:32:34.919497 21543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:32:34.926689 21543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:32:34.951399 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.939497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:38417 (local address 127.0.0.1:44372)
0929 16:32:34.940359 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:34.940402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:34.940593 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:34.943459 (+ 2866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:34.943471 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:34.943509 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:34.944044 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.944062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.946278 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.946284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.948544 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.948559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.950229 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.950235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.950491 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.950584 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 16:32:34.950742 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 16:32:34.950946 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":121,"thread_start_us":407,"threads_started":1}
I20260929 16:32:34.951501 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.940113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.5.232.131:38417)
0929 16:32:34.942720 (+ 2607us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:34.942733 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:34.942778 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:32:34.942897 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:34.942907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:34.943198 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:34.943604 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:34.944382 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.946129 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.948768 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.949995 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.950159 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.950420 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:34.950968 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 16:32:34.951153 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1985,"thread_start_us":1832,"threads_started":1}
I20260929 16:32:34.951577 22013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.939734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.0.0.1:36972)
0929 16:32:34.941845 (+ 2111us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:34.941872 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:34.942017 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:34.943175 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:34.943187 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:34.943231 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:34.944430 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.944443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.946379 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.946384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.948711 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.948725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.950703 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.950709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.950898 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.950930 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:32:34.951031 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 16:32:34.951164 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1954,"mutex_wait_us":5,"thread_start_us":1823,"threads_started":1}
I20260929 16:32:34.952498 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.940496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.5.232.129:41959)
0929 16:32:34.941250 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:34.941259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:34.942154 (+ 895us) server_negotiation.cc:407] Connection header received
0929 16:32:34.942268 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:34.942276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:34.942565 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:34.942793 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:34.944668 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.946224 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.948912 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.950448 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.950607 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.950849 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:34.951776 (+ 927us) server_negotiation.cc:299] Negotiation successful
0929 16:32:34.952200 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":451,"threads_started":1}
I20260929 16:32:34.953424 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.941995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45039 (local address 127.0.0.1:60352)
0929 16:32:34.944937 (+ 2942us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:34.944954 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:34.945077 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:34.945789 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:34.945794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:34.945819 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:34.946156 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.946165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.948177 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.948183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.950442 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:34.950456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.952608 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.952614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:34.952799 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.952829 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:32:34.952914 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:32:34.953056 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2813,"thread_start_us":2644,"threads_started":1}
I20260929 16:32:34.954157 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:34.942478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.5.232.130:45039)
0929 16:32:34.944437 (+ 1959us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:34.944450 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:34.945200 (+ 750us) server_negotiation.cc:407] Connection header received
0929 16:32:34.945301 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:34.945308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:34.945599 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:34.945810 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:34.946343 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.947989 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.950621 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:34.952135 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:34.952284 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:34.952463 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:34.953668 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0929 16:32:34.953848 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1736,"thread_start_us":1605,"threads_started":1}
I20260929 16:32:34.955929 21667 tablet_service.cc:1467] Processing CreateTablet for tablet fec374e73a1e4f26a06a4443b55d5d9c (DEFAULT_TABLE table=test-table [id=9c255c07d9a34eb88e2c3ae9891f3617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:34.955969 21934 tablet_service.cc:1467] Processing CreateTablet for tablet fec374e73a1e4f26a06a4443b55d5d9c (DEFAULT_TABLE table=test-table [id=9c255c07d9a34eb88e2c3ae9891f3617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:34.957257 21667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec374e73a1e4f26a06a4443b55d5d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:34.957471 21934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec374e73a1e4f26a06a4443b55d5d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:34.957950 21801 tablet_service.cc:1467] Processing CreateTablet for tablet fec374e73a1e4f26a06a4443b55d5d9c (DEFAULT_TABLE table=test-table [id=9c255c07d9a34eb88e2c3ae9891f3617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:34.959362 21801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec374e73a1e4f26a06a4443b55d5d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:34.969489 22018 tablet_bootstrap.cc:492] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: Bootstrap starting.
I20260929 16:32:34.972343 22019 tablet_bootstrap.cc:492] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: Bootstrap starting.
I20260929 16:32:34.974371 22018 tablet_bootstrap.cc:654] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:34.975739 22018 log.cc:824] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:34.976050 22019 tablet_bootstrap.cc:654] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:34.977320 22020 tablet_bootstrap.cc:492] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: Bootstrap starting.
I20260929 16:32:34.977421 22019 log.cc:824] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:34.978981 22018 tablet_bootstrap.cc:492] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: No bootstrap required, opened a new log
I20260929 16:32:34.979194 22018 ts_tablet_manager.cc:1397] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 16:32:34.982334 22020 tablet_bootstrap.cc:654] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:34.983867 22020 log.cc:824] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:34.986253 22019 tablet_bootstrap.cc:492] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: No bootstrap required, opened a new log
I20260929 16:32:34.986584 22019 ts_tablet_manager.cc:1397] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260929 16:32:34.987105 22020 tablet_bootstrap.cc:492] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: No bootstrap required, opened a new log
I20260929 16:32:34.987367 22020 ts_tablet_manager.cc:1397] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:32:34.988806 22018 raft_consensus.cc:348] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:34.989279 22018 raft_consensus.cc:374] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:34.989414 22018 raft_consensus.cc:729] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a0a340f85240b3a5581a550267ab0f, State: Initialized, Role: FOLLOWER
I20260929 16:32:34.990163 22018 consensus_queue.cc:260] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:34.992208 22018 ts_tablet_manager.cc:1428] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: Time spent starting tablet: real 0.013s user 0.001s sys 0.011s
I20260929 16:32:34.997776 22019 raft_consensus.cc:348] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:34.997951 22020 raft_consensus.cc:348] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:34.998298 22019 raft_consensus.cc:374] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:34.998402 22020 raft_consensus.cc:374] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:34.998407 22019 raft_consensus.cc:729] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642f2990e3b84ff7b6c6584fe57df64a, State: Initialized, Role: FOLLOWER
I20260929 16:32:34.998505 22020 raft_consensus.cc:729] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f75774c887b49b9820de2f25172ae5c, State: Initialized, Role: FOLLOWER
I20260929 16:32:34.999104 22020 consensus_queue.cc:260] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:34.999099 22019 consensus_queue.cc:260] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:35.000262 21999 heartbeater.cc:503] Master 127.5.232.190:42825 was elected leader, sending a full tablet report...
I20260929 16:32:35.000753 21866 heartbeater.cc:503] Master 127.5.232.190:42825 was elected leader, sending a full tablet report...
I20260929 16:32:35.001112 22019 ts_tablet_manager.cc:1428] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 16:32:35.001355 22020 ts_tablet_manager.cc:1428] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 16:32:35.057565 22026 raft_consensus.cc:482] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:32:35.057914 22026 raft_consensus.cc:504] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:35.059880 22026 leader_election.cc:288] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88a0a340f85240b3a5581a550267ab0f (127.5.232.129:41959), 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039)
I20260929 16:32:35.067593 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.060331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:50959 (local address 127.5.232.129:41959)
0929 16:32:35.060733 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:35.060741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:35.060770 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:32:35.060973 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:35.060980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:35.061086 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:35.061196 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:35.061680 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.062693 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.065326 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.066616 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.066791 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.066872 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:35.067238 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 16:32:35.067354 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260929 16:32:35.067660 22002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.060355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.5.232.131:50959)
0929 16:32:35.060600 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:35.060625 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:35.060769 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:35.061241 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:35.061249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:35.061286 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:35.061547 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:35.061556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.062825 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.062828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:35.065140 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:35.065157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.066803 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.066808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:35.067033 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.067076 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:32:35.067178 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:32:35.067328 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":82}
I20260929 16:32:35.067948 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.060480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:54407 (local address 127.5.232.130:45039)
0929 16:32:35.060756 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:35.060762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:35.061334 (+ 572us) server_negotiation.cc:407] Connection header received
0929 16:32:35.061506 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:35.061518 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:35.061576 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:35.061681 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:35.062149 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.063056 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.065580 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.066862 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.066997 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.067082 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:35.067530 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 16:32:35.067685 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260929 16:32:35.068434 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.060457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45039 (local address 127.5.232.131:54407)
0929 16:32:35.061214 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:35.061231 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:35.061357 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:35.061719 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:35.061722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:35.061743 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:35.062006 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:35.062013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.063309 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.063314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:35.065416 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:35.065432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.067191 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.067196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:35.067356 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.067392 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:32:35.067482 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:32:35.067608 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":458,"threads_started":1}
W20260929 16:32:35.068603 21733 tablet.cc:2369] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:35.068787 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec374e73a1e4f26a06a4443b55d5d9c" candidate_uuid: "642f2990e3b84ff7b6c6584fe57df64a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a0a340f85240b3a5581a550267ab0f" is_pre_election: true
I20260929 16:32:35.069439 21687 raft_consensus.cc:2454] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 642f2990e3b84ff7b6c6584fe57df64a in term 0.
I20260929 16:32:35.069607 21821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec374e73a1e4f26a06a4443b55d5d9c" candidate_uuid: "642f2990e3b84ff7b6c6584fe57df64a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f75774c887b49b9820de2f25172ae5c" is_pre_election: true
I20260929 16:32:35.070154 21821 raft_consensus.cc:2454] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 642f2990e3b84ff7b6c6584fe57df64a in term 0.
I20260929 16:32:35.070205 21889 leader_election.cc:302] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642f2990e3b84ff7b6c6584fe57df64a, 88a0a340f85240b3a5581a550267ab0f; no voters:
I20260929 16:32:35.070709 22026 raft_consensus.cc:2781] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:32:35.070865 22026 raft_consensus.cc:482] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:32:35.070976 22026 raft_consensus.cc:3037] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:35.073163 22026 raft_consensus.cc:504] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:35.073850 22026 leader_election.cc:288] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [CANDIDATE]: Term 1 election: Requested vote from peers 88a0a340f85240b3a5581a550267ab0f (127.5.232.129:41959), 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039)
I20260929 16:32:35.074368 21821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec374e73a1e4f26a06a4443b55d5d9c" candidate_uuid: "642f2990e3b84ff7b6c6584fe57df64a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f75774c887b49b9820de2f25172ae5c"
I20260929 16:32:35.074396 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec374e73a1e4f26a06a4443b55d5d9c" candidate_uuid: "642f2990e3b84ff7b6c6584fe57df64a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a0a340f85240b3a5581a550267ab0f"
I20260929 16:32:35.074569 21821 raft_consensus.cc:3037] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:35.074666 21687 raft_consensus.cc:3037] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:32:35.075464 21867 tablet.cc:2369] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:35.077078 21821 raft_consensus.cc:2454] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642f2990e3b84ff7b6c6584fe57df64a in term 1.
I20260929 16:32:35.077527 21687 raft_consensus.cc:2454] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 642f2990e3b84ff7b6c6584fe57df64a in term 1.
I20260929 16:32:35.077627 21890 leader_election.cc:302] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642f2990e3b84ff7b6c6584fe57df64a, 9f75774c887b49b9820de2f25172ae5c; no voters:
I20260929 16:32:35.077975 22026 raft_consensus.cc:2781] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:32:35.078439 22026 raft_consensus.cc:686] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 1 LEADER]: Becoming Leader. State: Replica: 642f2990e3b84ff7b6c6584fe57df64a, State: Running, Role: LEADER
I20260929 16:32:35.078995 22026 consensus_queue.cc:237] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:35.084709 21543 catalog_manager.cc:5495] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a reported cstate change: term changed from 0 to 1, leader changed from <none> to 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131). New cstate: current_term: 1 leader_uuid: "642f2990e3b84ff7b6c6584fe57df64a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } health_report { overall_health: UNKNOWN } } }
I20260929 16:32:35.109467 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.104109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.5.232.131:38417)
0929 16:32:35.104315 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:35.104320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:35.104340 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:32:35.104428 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:35.104431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:35.104523 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:35.104648 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:32:35.105056 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.105851 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.107331 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.107717 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.107807 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.108351 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:32:35.108867 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:32:35.108978 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:35.109074 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 16:32:35.109147 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 16:32:35.110814 22000 tablet.cc:2369] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:35.118880 21687 raft_consensus.cc:1261] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Refusing update from remote peer 642f2990e3b84ff7b6c6584fe57df64a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:35.118880 21821 raft_consensus.cc:1261] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [term 1 FOLLOWER]: Refusing update from remote peer 642f2990e3b84ff7b6c6584fe57df64a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:35.120064 22026 consensus_queue.cc:1035] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:35.120407 22028 consensus_queue.cc:1035] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:35.135535 22032 mvcc.cc:205] Tried to move back new op lower bound from 7334705377755095040 to 7334705377607696384. Current Snapshot: MvccSnapshot[applied={T|T < 7334705377755095040}]
I20260929 16:32:35.152685 21542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:35.153009 21542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:35.156893 21542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Sending DeleteTablet for 3 replicas of tablet fec374e73a1e4f26a06a4443b55d5d9c
I20260929 16:32:35.158157 21934 tablet_service.cc:1514] Processing DeleteTablet for tablet fec374e73a1e4f26a06a4443b55d5d9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:35 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:44372
I20260929 16:32:35.158227 21801 tablet_service.cc:1514] Processing DeleteTablet for tablet fec374e73a1e4f26a06a4443b55d5d9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:35 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:60352
I20260929 16:32:35.158200 21667 tablet_service.cc:1514] Processing DeleteTablet for tablet fec374e73a1e4f26a06a4443b55d5d9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:35 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:36972
I20260929 16:32:35.164404 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 21604
I20260929 16:32:35.186510 22041 tablet_replica.cc:330] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: stopping tablet replica
I20260929 16:32:35.187525 22041 raft_consensus.cc:2229] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:32:35.190524 22041 raft_consensus.cc:2258] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:35.191687 22043 tablet_replica.cc:330] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: stopping tablet replica
I20260929 16:32:35.192304 22043 raft_consensus.cc:2229] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:32:35.193280 22043 raft_consensus.cc:2258] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:35.202643 22042 tablet_replica.cc:330] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: stopping tablet replica
I20260929 16:32:35.203141 22042 raft_consensus.cc:2229] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:32:35.203330 22043 ts_tablet_manager.cc:1905] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:35.203727 22042 raft_consensus.cc:2258] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:35.204298 22041 ts_tablet_manager.cc:1905] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:35.206143 22042 ts_tablet_manager.cc:1905] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:35.210989 22043 ts_tablet_manager.cc:1918] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:35.211205 22043 log.cc:1196] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/fec374e73a1e4f26a06a4443b55d5d9c
I20260929 16:32:35.211542 22041 ts_tablet_manager.cc:1918] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:35.211726 22041 log.cc:1196] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/fec374e73a1e4f26a06a4443b55d5d9c
I20260929 16:32:35.211767 22043 ts_tablet_manager.cc:1939] T fec374e73a1e4f26a06a4443b55d5d9c P 642f2990e3b84ff7b6c6584fe57df64a: Deleting consensus metadata
I20260929 16:32:35.212252 22041 ts_tablet_manager.cc:1939] T fec374e73a1e4f26a06a4443b55d5d9c P 9f75774c887b49b9820de2f25172ae5c: Deleting consensus metadata
I20260929 16:32:35.213904 22042 ts_tablet_manager.cc:1918] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:35.213858 21528 catalog_manager.cc:4841] TS 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417): tablet fec374e73a1e4f26a06a4443b55d5d9c (table test-table [id=9c255c07d9a34eb88e2c3ae9891f3617]) successfully deleted
I20260929 16:32:35.214179 22042 log.cc:1196] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/fec374e73a1e4f26a06a4443b55d5d9c
I20260929 16:32:35.214870 22042 ts_tablet_manager.cc:1939] T fec374e73a1e4f26a06a4443b55d5d9c P 88a0a340f85240b3a5581a550267ab0f: Deleting consensus metadata
I20260929 16:32:35.215077 21530 catalog_manager.cc:4841] TS 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039): tablet fec374e73a1e4f26a06a4443b55d5d9c (table test-table [id=9c255c07d9a34eb88e2c3ae9891f3617]) successfully deleted
I20260929 16:32:35.216704 21529 catalog_manager.cc:4841] TS 88a0a340f85240b3a5581a550267ab0f (127.5.232.129:41959): tablet fec374e73a1e4f26a06a4443b55d5d9c (table test-table [id=9c255c07d9a34eb88e2c3ae9891f3617]) successfully deleted
Sep 29 16:32:35 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:32:35.225049 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.166174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36988 (local address 127.5.232.129:41959)
0929 16:32:35.166425 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:35.166434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:35.181828 (+ 15394us) server_negotiation.cc:407] Connection header received
0929 16:32:35.182742 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:35.182753 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:35.182877 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:35.182971 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:35.187863 (+ 4892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.189672 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.199773 (+ 10101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.200867 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.201022 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.218717 (+ 17695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:35.218730 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:35.218745 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:35.218789 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:35.221741 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:35.222857 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:35.222866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:35.222868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:35.223016 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:35.223480 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:35.223483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:35.223485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:35.223926 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:35.224153 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:35.224550 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 16:32:35.224729 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:32:35.225862 21727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:35.317718 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 21604
I20260929 16:32:35.335830 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 21738
Sep 29 16:32:35 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:32:35.353168 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.336896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.5.232.130:45039)
0929 16:32:35.337152 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:35.337156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:35.337372 (+ 216us) server_negotiation.cc:407] Connection header received
0929 16:32:35.337654 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:35.337662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:35.337747 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:35.337860 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:35.339332 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.340132 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.341632 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.342057 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.342203 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.346730 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:35.346745 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:35.346760 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:35.346801 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:35.350161 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:35.350975 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:35.350984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:35.350986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:35.351113 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:35.351536 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:35.351540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:35.351542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:35.351938 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:35.352246 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:35.352590 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 16:32:35.352820 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260929 16:32:35.354045 21861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:35.445238 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 21738
I20260929 16:32:35.462899 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 21871
Sep 29 16:32:35 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:32:35.481894 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:35.463689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.5.232.131:38417)
0929 16:32:35.463864 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:35.463869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:35.463894 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:32:35.464119 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:35.464123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:35.464207 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:35.464288 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:35.466238 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.467003 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.469065 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:35.469723 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:35.469832 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:35.475386 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:35.475400 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:35.475420 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:35.475467 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:35.478807 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:35.479571 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:35.479579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:35.479581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:35.479772 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:35.480194 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:35.480198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:35.480200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:35.480588 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:35.480919 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:35.481240 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 16:32:35.481498 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:32:35.482653 21994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:35.608063 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 21871
I20260929 16:32:35.628224 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 21511
I20260929 16:32:35.629369 21573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:35.746891 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 21511
I20260929 16:32:55.768610 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42825
--webserver_interface=127.5.232.190
--webserver_port=46425
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:32:56.106042 22049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:56.106354 22049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:56.106544 22049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:56.114017 22049 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:32:56.114122 22049 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:32:56.114203 22049 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:32:56.114310 22049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:32:56.114403 22049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:56.114523 22049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:56.114602 22049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:32:56.114686 22049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:32:56.124867 22049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=46425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:56.126232 22049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:56.127673 22049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:56.151681 22055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:56.151710 22056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:56.151767 22058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:56.152194 22049 server_base.cc:1034] running on GCE node
I20260929 16:32:56.153412 22049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:56.155408 22049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:56.156733 22049 hybrid_clock.cc:648] HybridClock initialized: now 1790699576156573 us; error 109 us; skew 500 ppm
I20260929 16:32:56.157167 22049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:56 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:56.162554 22049 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:32:56.164634 22049 webserver.cc:460] Webserver started at http://127.5.232.190:46425/ using document root <none> and password file <none>
I20260929 16:32:56.165666 22049 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:56.165778 22049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:56.172048 22049 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 16:32:56.175168 22065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:56.176677 22049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:32:56.176831 22049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "a386273cf011d27e015bb7838c25d23b"
server_key_iv: "720ff0ea643584211674f976f68f2b19"
server_key_version: "encryptionkey@0"
I20260929 16:32:56.177979 22049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:56.199546 22049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:56.204140 22049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:56.204414 22049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:56.224114 22049 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:42825
I20260929 16:32:56.224138 22116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:42825 every 8 connection(s)
I20260929 16:32:56.225834 22049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:32:56.233139 22117 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:32:56.234011 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 22049
I20260929 16:32:56.235067 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:41959
--local_ip_for_outbound_sockets=127.5.232.129
--tserver_master_addrs=127.5.232.190:42825
--webserver_port=34527
--webserver_interface=127.5.232.129
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:32:56.238709 22117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Bootstrap starting.
I20260929 16:32:56.260584 22117 log.cc:824] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:56.273221 22117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:32:56.273854 22117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Bootstrap complete.
I20260929 16:32:56.283782 22117 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:32:56.284737 22117 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7eaf0bd0d94a6daf8b8041e7f10ed8, State: Initialized, Role: FOLLOWER
I20260929 16:32:56.285508 22117 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:32:56.285764 22117 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:32:56.285885 22117 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:32:56.286077 22117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:32:56.290197 22117 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:32:56.290798 22117 leader_election.cc:302] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf7eaf0bd0d94a6daf8b8041e7f10ed8; no voters:
I20260929 16:32:56.291502 22117 leader_election.cc:288] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:32:56.291651 22121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:32:56.292788 22121 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 2 LEADER]: Becoming Leader. State: Replica: bf7eaf0bd0d94a6daf8b8041e7f10ed8, State: Running, Role: LEADER
I20260929 16:32:56.293465 22121 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:32:56.293998 22117 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:32:56.296263 22123 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf7eaf0bd0d94a6daf8b8041e7f10ed8. Latest consensus state: current_term: 2 leader_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } } }
I20260929 16:32:56.296659 22123 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:56.296319 22122 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } } }
I20260929 16:32:56.296770 22122 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:32:56.306977 22127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:32:56.317474 22127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:32:56.319494 22127 catalog_manager.cc:1259] Loaded cluster ID: b26230192ffb438396ce0d22c645caf5
I20260929 16:32:56.319700 22127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:32:56.325316 22127 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:32:56.328066 22127 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Loaded TSK: 0
I20260929 16:32:56.334903 22138 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:32:56.351907 22127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Generated new TSK 1
I20260929 16:32:56.352300 22127 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:32:56.617168 22119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:56.617585 22119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:56.617741 22119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:56.625152 22119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:56.625334 22119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:56.625538 22119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:32:56.635672 22119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:41959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=34527
--tserver_master_addrs=127.5.232.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:56.637061 22119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:56.638511 22119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:56.653971 22144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:56.654029 22147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:56.654238 22145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:56.656045 22119 server_base.cc:1034] running on GCE node
I20260929 16:32:56.656762 22119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:56.657871 22119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:56.659119 22119 hybrid_clock.cc:648] HybridClock initialized: now 1790699576659060 us; error 57 us; skew 500 ppm
I20260929 16:32:56.659471 22119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:56 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:56.662940 22119 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:32:56.664562 22119 webserver.cc:460] Webserver started at http://127.5.232.129:34527/ using document root <none> and password file <none>
I20260929 16:32:56.665328 22119 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:56.665520 22119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:56.671310 22119 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 16:32:56.674849 22154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:56.676388 22119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:32:56.676596 22119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "88a0a340f85240b3a5581a550267ab0f"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "90cb1422341932745693163b61e0f2f2"
server_key_iv: "aed754f53b1f5659ec3e733d305cdb83"
server_key_version: "encryptionkey@0"
I20260929 16:32:56.677696 22119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:56.703568 22119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:56.708226 22119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:56.708547 22119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:56.709640 22119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:56.711045 22119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:56.711169 22119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:56.711304 22119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:56.711403 22119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:56.745024 22119 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:41959
I20260929 16:32:56.745114 22266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:41959 every 8 connection(s)
I20260929 16:32:56.746582 22119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:32:56.753969 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 22119
I20260929 16:32:56.754622 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:45039
--local_ip_for_outbound_sockets=127.5.232.130
--tserver_master_addrs=127.5.232.190:42825
--webserver_port=39037
--webserver_interface=127.5.232.130
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:32:56 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:56.771435 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:56.749494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:55365)
0929 16:32:56.750312 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:56.750358 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:56.752104 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:56.753527 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:56.753535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:56.754100 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:56.754666 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:56.754683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:56.756442 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:56.756446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:56.757793 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:56.757807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:56.758901 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:56.758909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:56.759198 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:56.759919 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:56.759941 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:56.762217 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:56.767078 (+ 4861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:56.767086 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:56.767100 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:56.767567 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:56.768096 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:56.768099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:56.768102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:56.768311 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:56.769319 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:56.769325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:56.769741 (+ 416us) client_negotiation.cc:769] Sending connection context
0929 16:32:56.769971 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 16:32:56.770405 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":356,"threads_started":1}
I20260929 16:32:56.772040 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:56.749338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:55365 (local address 127.5.232.190:42825)
0929 16:32:56.750010 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:56.750021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:56.750479 (+ 458us) server_negotiation.cc:407] Connection header received
0929 16:32:56.752608 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:56.752626 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:56.752983 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:56.753313 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:56.754958 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:56.756249 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:56.757958 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:56.758642 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:56.758864 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:56.762645 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:56.762659 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:56.762682 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:56.762737 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:56.766886 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:56.767738 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:56.767751 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:56.767762 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:56.767980 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:56.768450 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:56.768458 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:56.768464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:56.769185 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:56.769414 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:56.770271 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 16:32:56.770769 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":273,"threads_started":1}
I20260929 16:32:56.775120 22267 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42825
I20260929 16:32:56.775468 22267 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:56.776206 22267 heartbeater.cc:511] Master 127.5.232.190:42825 requested a full tablet report, sending...
I20260929 16:32:56.778560 22082 ts_manager.cc:194] Registered new tserver with Master: 88a0a340f85240b3a5581a550267ab0f (127.5.232.129:41959)
I20260929 16:32:56.780079 22082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:55365
W20260929 16:32:57.088750 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:57.089104 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:57.089280 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:57.096729 22271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:57.096890 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:57.097093 22271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:32:57.107148 22271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:45039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=39037
--tserver_master_addrs=127.5.232.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:57.108553 22271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:57.110041 22271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:57.123721 22276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:57.123760 22277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:57.123793 22279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:57.130087 22271 server_base.cc:1034] running on GCE node
I20260929 16:32:57.130757 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:57.131748 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:57.132961 22271 hybrid_clock.cc:648] HybridClock initialized: now 1790699577132897 us; error 63 us; skew 500 ppm
I20260929 16:32:57.133246 22271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:57 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699577, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:57.136390 22271 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:32:57.138103 22271 webserver.cc:460] Webserver started at http://127.5.232.130:39037/ using document root <none> and password file <none>
I20260929 16:32:57.138855 22271 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:57.139034 22271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:57.144580 22271 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:32:57.147372 22286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:57.148687 22271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:32:57.148867 22271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "9f75774c887b49b9820de2f25172ae5c"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "3f1b4c94aa21aa47c4e9d9980c07764d"
server_key_iv: "bc0e02312fdc84051c7907aebd7bdd30"
server_key_version: "encryptionkey@0"
I20260929 16:32:57.149909 22271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:57.176357 22271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:57.180485 22271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:57.180804 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:57.182013 22271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:57.183460 22271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:57.183589 22271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:57.183725 22271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:57.183809 22271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:57.217352 22271 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45039
I20260929 16:32:57.217433 22398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45039 every 8 connection(s)
I20260929 16:32:57.218852 22271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:32:57.222051 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 22271
I20260929 16:32:57.222832 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:38417
--local_ip_for_outbound_sockets=127.5.232.131
--tserver_master_addrs=127.5.232.190:42825
--webserver_port=41165
--webserver_interface=127.5.232.131
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:32:57 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699577, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:57.245019 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.221432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:47953 (local address 127.5.232.190:42825)
0929 16:32:57.221626 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.221631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.223013 (+ 1382us) server_negotiation.cc:407] Connection header received
0929 16:32:57.225676 (+ 2663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.225684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.225801 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.225914 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:57.227859 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.229065 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.231453 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.232272 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.232426 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.237048 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:57.237057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:57.237066 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:57.237117 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:57.241156 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:57.242208 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:57.242217 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:57.242224 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:57.242385 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:57.242931 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:57.242938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:57.242945 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:57.243391 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:57.243579 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:57.244262 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 16:32:57.244569 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:32:57.245666 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.222055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:47953)
0929 16:32:57.222824 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:57.222861 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:57.225318 (+ 2457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:57.226182 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:57.226194 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:57.226855 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:57.227648 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.227674 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.229240 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.229245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.231275 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.231289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.232508 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.232515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.232802 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.233702 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:57.233733 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:57.236724 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:57.241340 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:57.241350 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:57.241369 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:57.242043 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:57.242530 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:57.242534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:57.242537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:57.242791 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:57.243535 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:57.243542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:57.243873 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 16:32:57.244103 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 16:32:57.244509 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":314,"threads_started":1}
I20260929 16:32:57.247303 22399 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42825
I20260929 16:32:57.247625 22399 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:57.248410 22399 heartbeater.cc:511] Master 127.5.232.190:42825 requested a full tablet report, sending...
I20260929 16:32:57.249848 22082 ts_manager.cc:194] Registered new tserver with Master: 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039)
I20260929 16:32:57.250646 22082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:47953
W20260929 16:32:57.556308 22402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:32:57.556663 22402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:32:57.556833 22402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:32:57.564330 22402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:32:57.564509 22402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:32:57.564698 22402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:32:57.574668 22402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:38417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=41165
--tserver_master_addrs=127.5.232.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:32:57.576059 22402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:32:57.577481 22402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:32:57.590957 22410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:57.591078 22408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:32:57.591053 22407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:32:57.591900 22402 server_base.cc:1034] running on GCE node
I20260929 16:32:57.592550 22402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:32:57.593601 22402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:32:57.594826 22402 hybrid_clock.cc:648] HybridClock initialized: now 1790699577594775 us; error 46 us; skew 500 ppm
I20260929 16:32:57.595162 22402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:32:57 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699577, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:32:57.598417 22402 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:32:57.600174 22402 webserver.cc:460] Webserver started at http://127.5.232.131:41165/ using document root <none> and password file <none>
I20260929 16:32:57.600880 22402 fs_manager.cc:359] Metadata directory not provided
I20260929 16:32:57.601006 22402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:32:57.606509 22402 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:32:57.609587 22417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:57.611054 22402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:32:57.611244 22402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "642f2990e3b84ff7b6c6584fe57df64a"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "788e8cb8ff5274699380a7881682620b"
server_key_iv: "ece20ef5d0418be1d9af8bd3c8451b3c"
server_key_version: "encryptionkey@0"
I20260929 16:32:57.612248 22402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:32:57.629509 22402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:32:57.633692 22402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:32:57.634038 22402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:32:57.635105 22402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:32:57.636459 22402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:32:57.636544 22402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:57.636677 22402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:32:57.636793 22402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:32:57.670485 22402 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:38417
I20260929 16:32:57.670550 22529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:38417 every 8 connection(s)
I20260929 16:32:57.671957 22402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:32:57.679346 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 22402
Sep 29 16:32:57 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699577, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:32:57.693333 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.674705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:54521 (local address 127.5.232.190:42825)
0929 16:32:57.674899 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.674904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.676006 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 16:32:57.677897 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.677903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.677992 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.678115 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:57.679719 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.680911 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.683012 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.683425 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.683588 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.687489 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:57.687494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:57.687497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:57.687528 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:57.690141 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:57.690935 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:57.690941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:57.690946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:57.691098 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:57.691560 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:57.691564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:57.691566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:57.691836 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:57.692002 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:57.692723 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 16:32:57.692954 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:32:57.694644 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.675141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:54521)
0929 16:32:57.675835 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:57.675875 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:57.677613 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:57.678377 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:57.678387 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:57.678901 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:32:57.679546 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.679565 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.681097 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.681105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.682867 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.682879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.683653 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.683658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.683872 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.684587 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:32:57.684619 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:32:57.687271 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:32:57.690328 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:57.690337 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:57.690356 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:57.690804 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:57.691216 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:32:57.691223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:32:57.691227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:32:57.691425 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:32:57.691972 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:32:57.691980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:32:57.692349 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 16:32:57.692625 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 16:32:57.693194 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":281,"threads_started":1}
I20260929 16:32:57.695726 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.680344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.5.232.190:42825)
0929 16:32:57.681534 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.681539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.681556 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:32:57.681603 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.681608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.681672 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.681780 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:57.683514 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.684496 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.686445 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.686968 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.687114 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.690263 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:57.690269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:57.690272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:57.690302 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:57.692833 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:57.693614 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:57.693618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:57.693621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:57.693809 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:57.694236 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:57.694239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:57.694241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:57.694484 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:57.694672 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:57.695102 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 16:32:57.695319 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":924,"threads_started":1}
I20260929 16:32:57.696292 22530 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42825
I20260929 16:32:57.696610 22530 heartbeater.cc:464] Registering TS with master...
I20260929 16:32:57.697683 22530 heartbeater.cc:511] Master 127.5.232.190:42825 requested a full tablet report, sending...
I20260929 16:32:57.699211 22082 ts_manager.cc:194] Registered new tserver with Master: 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417)
I20260929 16:32:57.700037 22082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:54521
I20260929 16:32:57.710958 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:32:57.717250 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.711954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.5.232.190:42825)
0929 16:32:57.712190 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.712197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.712409 (+ 212us) server_negotiation.cc:407] Connection header received
0929 16:32:57.712588 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.712591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.712729 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.712855 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:32:57.713226 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.713997 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.715419 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.715809 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.715946 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.716423 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:32:57.716629 (+ 206us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 16:32:57.716928 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41686: token expired
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 16:32:57.717427 22534 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41686: token expired
W20260929 16:32:57.717911 22003 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.232.190:42825)
I20260929 16:32:57.731628 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.718337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.5.232.190:42825)
0929 16:32:57.718494 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.718501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.718519 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:32:57.718652 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.718659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.718724 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.718799 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:57.720121 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.720925 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.722306 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.722717 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.722862 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.725707 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:57.725714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:57.725719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:57.725781 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:57.729014 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:57.729864 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:57.729869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:57.729871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:57.729999 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:57.730322 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:57.730325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:57.730327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:57.730655 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:57.730834 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:57.731109 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 16:32:57.731280 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:32:57.735157 22082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:32:57.742722 22082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:32:57.778462 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.755405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.0.0.1:50440)
0929 16:32:57.769848 (+ 14443us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:57.769899 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:57.770112 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:57.771089 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:57.771100 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:57.771148 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:57.771731 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.771744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.773545 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.773551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.775695 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.775706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.777451 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.777457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.777663 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.777716 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 16:32:57.777861 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:32:57.778061 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14190,"mutex_wait_us":26,"thread_start_us":15361,"threads_started":1}
I20260929 16:32:57.778692 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.755647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.5.232.129:41959)
0929 16:32:57.768929 (+ 13282us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.768942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.770345 (+ 1403us) server_negotiation.cc:407] Connection header received
0929 16:32:57.770461 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.770470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.770751 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.770948 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:57.771929 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.773358 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.775870 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.776827 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.776997 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.777258 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:57.778165 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 16:32:57.778360 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13033,"thread_start_us":13324,"threads_started":1}
I20260929 16:32:57.780113 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.756877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.5.232.131:38417)
0929 16:32:57.764805 (+ 7928us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.764817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.771166 (+ 6349us) server_negotiation.cc:407] Connection header received
0929 16:32:57.771908 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.771916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.772152 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.772331 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:57.772956 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.774515 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.777134 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.778599 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.778715 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.778954 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:57.779632 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 16:32:57.779789 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7758,"thread_start_us":7586,"threads_started":1}
I20260929 16:32:57.780115 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.757493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.5.232.130:45039)
0929 16:32:57.764122 (+ 6629us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.764135 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.771747 (+ 7612us) server_negotiation.cc:407] Connection header received
0929 16:32:57.771967 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.771974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.772149 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.772309 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:57.773189 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.774796 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.777272 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.778657 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.778777 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.778954 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:57.779615 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 16:32:57.779795 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6390,"thread_start_us":6200,"threads_started":1}
I20260929 16:32:57.780896 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.756095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45039 (local address 127.0.0.1:47972)
0929 16:32:57.771606 (+ 15511us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:57.771620 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:57.771779 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:57.772632 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:57.772635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:57.772657 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:57.773009 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.773021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.774937 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.774944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.777101 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.777114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.779147 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.779161 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.779359 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.779445 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 16:32:57.779557 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:32:57.779686 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15405,"mutex_wait_us":48,"thread_start_us":13342,"threads_started":1}
I20260929 16:32:57.780949 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.755666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:38417 (local address 127.0.0.1:55226)
0929 16:32:57.771038 (+ 15372us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:57.771055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:57.771666 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:57.772353 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:57.772358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:57.772384 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:57.772729 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.772744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.774665 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.774671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.776934 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.776956 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.779174 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.779179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.779428 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.779465 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:32:57.780145 (+ 680us) client_negotiation.cc:240] Negotiation successful
0929 16:32:57.780516 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15235,"mutex_wait_us":42,"thread_start_us":15693,"threads_started":1}
I20260929 16:32:57.783125 22202 tablet_service.cc:1467] Processing CreateTablet for tablet 227f9324e45942cc8ad71215d02bab67 (DEFAULT_TABLE table=test-table [id=30035f6337b842238c73b78b25cd2038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:57.783982 22267 heartbeater.cc:503] Master 127.5.232.190:42825 was elected leader, sending a full tablet report...
I20260929 16:32:57.784967 22202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227f9324e45942cc8ad71215d02bab67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:57.784888 22334 tablet_service.cc:1467] Processing CreateTablet for tablet 227f9324e45942cc8ad71215d02bab67 (DEFAULT_TABLE table=test-table [id=30035f6337b842238c73b78b25cd2038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:57.785501 22465 tablet_service.cc:1467] Processing CreateTablet for tablet 227f9324e45942cc8ad71215d02bab67 (DEFAULT_TABLE table=test-table [id=30035f6337b842238c73b78b25cd2038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:32:57.786187 22334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227f9324e45942cc8ad71215d02bab67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:57.786793 22465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227f9324e45942cc8ad71215d02bab67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:32:57.798682 22546 tablet_bootstrap.cc:492] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: Bootstrap starting.
I20260929 16:32:57.802886 22546 tablet_bootstrap.cc:654] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:57.803958 22546 log.cc:824] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:57.806085 22548 tablet_bootstrap.cc:492] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: Bootstrap starting.
I20260929 16:32:57.807794 22546 tablet_bootstrap.cc:492] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: No bootstrap required, opened a new log
I20260929 16:32:57.808104 22546 ts_tablet_manager.cc:1397] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:32:57.809607 22547 tablet_bootstrap.cc:492] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Bootstrap starting.
I20260929 16:32:57.809960 22548 tablet_bootstrap.cc:654] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:57.811219 22548 log.cc:824] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:57.813138 22547 tablet_bootstrap.cc:654] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Neither blocks nor log segments found. Creating new log.
I20260929 16:32:57.814203 22547 log.cc:824] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:32:57.814638 22548 tablet_bootstrap.cc:492] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: No bootstrap required, opened a new log
I20260929 16:32:57.814904 22548 ts_tablet_manager.cc:1397] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:32:57.817344 22547 tablet_bootstrap.cc:492] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: No bootstrap required, opened a new log
I20260929 16:32:57.817634 22547 ts_tablet_manager.cc:1397] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260929 16:32:57.818516 22546 raft_consensus.cc:348] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:57.818980 22546 raft_consensus.cc:374] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:57.819085 22546 raft_consensus.cc:729] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f75774c887b49b9820de2f25172ae5c, State: Initialized, Role: FOLLOWER
I20260929 16:32:57.819723 22546 consensus_queue.cc:260] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:57.822412 22399 heartbeater.cc:503] Master 127.5.232.190:42825 was elected leader, sending a full tablet report...
I20260929 16:32:57.823066 22546 ts_tablet_manager.cc:1428] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260929 16:32:57.826084 22548 raft_consensus.cc:348] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:57.826539 22548 raft_consensus.cc:374] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:57.826674 22548 raft_consensus.cc:729] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642f2990e3b84ff7b6c6584fe57df64a, State: Initialized, Role: FOLLOWER
I20260929 16:32:57.826634 22547 raft_consensus.cc:348] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:57.827042 22547 raft_consensus.cc:374] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:32:57.827144 22547 raft_consensus.cc:729] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a0a340f85240b3a5581a550267ab0f, State: Initialized, Role: FOLLOWER
I20260929 16:32:57.827332 22548 consensus_queue.cc:260] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:57.827728 22547 consensus_queue.cc:260] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:57.828568 22530 heartbeater.cc:503] Master 127.5.232.190:42825 was elected leader, sending a full tablet report...
I20260929 16:32:57.830232 22548 ts_tablet_manager.cc:1428] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260929 16:32:57.830746 22547 ts_tablet_manager.cc:1428] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
W20260929 16:32:57.923712 22531 tablet.cc:2369] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:57.934347 22552 raft_consensus.cc:482] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:32:57.934662 22552 raft_consensus.cc:504] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:57.936378 22552 leader_election.cc:288] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88a0a340f85240b3a5581a550267ab0f (127.5.232.129:41959), 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417)
I20260929 16:32:57.945945 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.936857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:35083 (local address 127.5.232.131:38417)
0929 16:32:57.937060 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.937066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.937929 (+ 863us) server_negotiation.cc:407] Connection header received
0929 16:32:57.938135 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.938140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.938220 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.938325 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:57.939150 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.940257 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.943406 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.944662 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.944843 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.944929 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:57.945535 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 16:32:57.945674 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:32:57.945938 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.936944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:48203 (local address 127.5.232.129:41959)
0929 16:32:57.937136 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:57.937141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:57.937953 (+ 812us) server_negotiation.cc:407] Connection header received
0929 16:32:57.938135 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:57.938140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:57.938219 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:57.938325 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:32:57.939150 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.940257 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.943405 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.944669 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.944848 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.944921 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:57.945568 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 16:32:57.945694 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:32:57.946157 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.936870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:38417 (local address 127.5.232.130:35083)
0929 16:32:57.937754 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:57.937774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:57.937963 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:57.938363 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:57.938368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:57.938428 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:57.939012 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.939024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.940411 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.940416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.943231 (+ 2815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.943248 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.944941 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.944947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.945242 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.945353 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 16:32:57.945478 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 16:32:57.945627 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":323,"spinlock_wait_cycles":52096,"thread_start_us":519,"threads_started":1}
I20260929 16:32:57.946661 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:57.936866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.5.232.130:48203)
0929 16:32:57.937837 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:57.937855 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:32:57.938005 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:32:57.938367 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:32:57.938370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:32:57.938417 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:32:57.938915 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.938936 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.940410 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.940416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.943236 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:32:57.943248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:32:57.944942 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:32:57.944947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:32:57.945240 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:57.945353 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 16:32:57.945518 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 16:32:57.945626 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":346,"spinlock_wait_cycles":26240,"thread_start_us":426,"threads_started":1}
I20260929 16:32:57.947223 22485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227f9324e45942cc8ad71215d02bab67" candidate_uuid: "9f75774c887b49b9820de2f25172ae5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642f2990e3b84ff7b6c6584fe57df64a" is_pre_election: true
I20260929 16:32:57.947736 22485 raft_consensus.cc:2454] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f75774c887b49b9820de2f25172ae5c in term 0.
I20260929 16:32:57.947793 22222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227f9324e45942cc8ad71215d02bab67" candidate_uuid: "9f75774c887b49b9820de2f25172ae5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a0a340f85240b3a5581a550267ab0f" is_pre_election: true
I20260929 16:32:57.948369 22222 raft_consensus.cc:2454] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f75774c887b49b9820de2f25172ae5c in term 0.
I20260929 16:32:57.948480 22288 leader_election.cc:302] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642f2990e3b84ff7b6c6584fe57df64a, 9f75774c887b49b9820de2f25172ae5c; no voters:
I20260929 16:32:57.948958 22552 raft_consensus.cc:2781] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:32:57.949111 22552 raft_consensus.cc:482] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:32:57.949216 22552 raft_consensus.cc:3037] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:57.951459 22552 raft_consensus.cc:504] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:57.952127 22552 leader_election.cc:288] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [CANDIDATE]: Term 1 election: Requested vote from peers 88a0a340f85240b3a5581a550267ab0f (127.5.232.129:41959), 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417)
I20260929 16:32:57.952589 22222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227f9324e45942cc8ad71215d02bab67" candidate_uuid: "9f75774c887b49b9820de2f25172ae5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a0a340f85240b3a5581a550267ab0f"
I20260929 16:32:57.952622 22485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227f9324e45942cc8ad71215d02bab67" candidate_uuid: "9f75774c887b49b9820de2f25172ae5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642f2990e3b84ff7b6c6584fe57df64a"
I20260929 16:32:57.952790 22222 raft_consensus.cc:3037] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:57.952870 22485 raft_consensus.cc:3037] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:32:57.955253 22222 raft_consensus.cc:2454] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f75774c887b49b9820de2f25172ae5c in term 1.
I20260929 16:32:57.955281 22485 raft_consensus.cc:2454] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f75774c887b49b9820de2f25172ae5c in term 1.
I20260929 16:32:57.955673 22289 leader_election.cc:302] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a0a340f85240b3a5581a550267ab0f, 9f75774c887b49b9820de2f25172ae5c; no voters:
I20260929 16:32:57.956009 22552 raft_consensus.cc:2781] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:32:57.956422 22552 raft_consensus.cc:686] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 1 LEADER]: Becoming Leader. State: Replica: 9f75774c887b49b9820de2f25172ae5c, State: Running, Role: LEADER
I20260929 16:32:57.956894 22552 consensus_queue.cc:237] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:32:57.962180 22082 catalog_manager.cc:5495] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f75774c887b49b9820de2f25172ae5c (127.5.232.130). New cstate: current_term: 1 leader_uuid: "9f75774c887b49b9820de2f25172ae5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } health_report { overall_health: HEALTHY } } }
W20260929 16:32:57.970790 22400 tablet.cc:2369] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:32:57.999010 22268 tablet.cc:2369] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:32:58.025082 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.019259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.5.232.130:45039)
0929 16:32:58.019496 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:58.019505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:58.019760 (+ 255us) server_negotiation.cc:407] Connection header received
0929 16:32:58.020040 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:58.020045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:58.020127 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:58.020296 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:32:58.020658 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:58.021421 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:58.022810 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:58.023214 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:58.023362 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:58.023921 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:32:58.024479 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:32:58.024655 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:58.024720 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 16:32:58.024803 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:32:58.033540 22222 raft_consensus.cc:1261] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Refusing update from remote peer 9f75774c887b49b9820de2f25172ae5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:58.033540 22485 raft_consensus.cc:1261] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [term 1 FOLLOWER]: Refusing update from remote peer 9f75774c887b49b9820de2f25172ae5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:32:58.034586 22557 consensus_queue.cc:1035] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:58.034935 22552 consensus_queue.cc:1035] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:32:58.050998 22562 mvcc.cc:205] Tried to move back new op lower bound from 7334705471614377984 to 7334705471315562496. Current Snapshot: MvccSnapshot[applied={T|T < 7334705471614377984}]
I20260929 16:32:58.060499 22082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:58.060738 22082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:32:58.064353 22082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Sending DeleteTablet for 3 replicas of tablet 227f9324e45942cc8ad71215d02bab67
I20260929 16:32:58.065547 22202 tablet_service.cc:1514] Processing DeleteTablet for tablet 227f9324e45942cc8ad71215d02bab67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:58 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:50440
I20260929 16:32:58.065563 22465 tablet_service.cc:1514] Processing DeleteTablet for tablet 227f9324e45942cc8ad71215d02bab67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:58 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:55226
I20260929 16:32:58.065665 22334 tablet_service.cc:1514] Processing DeleteTablet for tablet 227f9324e45942cc8ad71215d02bab67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:32:58 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:47972
I20260929 16:32:58.066658 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 22119
I20260929 16:32:58.077610 22571 tablet_replica.cc:330] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: stopping tablet replica
I20260929 16:32:58.078374 22571 raft_consensus.cc:2229] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:32:58.079201 22571 raft_consensus.cc:2258] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:58.080188 22570 tablet_replica.cc:330] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: stopping tablet replica
I20260929 16:32:58.080938 22572 tablet_replica.cc:330] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: stopping tablet replica
I20260929 16:32:58.081496 22572 raft_consensus.cc:2229] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:32:58.082355 22572 raft_consensus.cc:2258] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:58.082609 22571 ts_tablet_manager.cc:1905] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:58.084383 22572 ts_tablet_manager.cc:1905] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:32:58.089577 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.068246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.5.232.129:41959)
0929 16:32:58.068451 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:58.068457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:58.068475 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:32:58.068535 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:58.068543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:58.068620 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:58.068723 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:58.070782 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:58.071953 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:58.073959 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:58.074638 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:58.074765 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:58.080046 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:58.080067 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:58.080085 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:58.080138 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:58.085044 (+ 4906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:58.086135 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:58.086145 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:58.086149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:58.086325 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:58.086859 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:58.086864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:58.086866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:58.087363 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:58.087669 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:58.088914 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0929 16:32:58.089148 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:32:58.089715 22571 ts_tablet_manager.cc:1918] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:58.089872 22571 log.cc:1196] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/227f9324e45942cc8ad71215d02bab67
I20260929 16:32:58.090279 22262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:58.090435 22571 ts_tablet_manager.cc:1939] T 227f9324e45942cc8ad71215d02bab67 P 642f2990e3b84ff7b6c6584fe57df64a: Deleting consensus metadata
I20260929 16:32:58.092361 22067 catalog_manager.cc:4841] TS 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417): tablet 227f9324e45942cc8ad71215d02bab67 (table test-table [id=30035f6337b842238c73b78b25cd2038]) successfully deleted
I20260929 16:32:58.092806 22572 ts_tablet_manager.cc:1918] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:32:58.092998 22572 log.cc:1196] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/227f9324e45942cc8ad71215d02bab67
I20260929 16:32:58.093595 22572 ts_tablet_manager.cc:1939] T 227f9324e45942cc8ad71215d02bab67 P 9f75774c887b49b9820de2f25172ae5c: Deleting consensus metadata
I20260929 16:32:58.095227 22069 catalog_manager.cc:4841] TS 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039): tablet 227f9324e45942cc8ad71215d02bab67 (table test-table [id=30035f6337b842238c73b78b25cd2038]) successfully deleted
I20260929 16:32:58.092517 22570 raft_consensus.cc:2229] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:32:58.188475 22570 raft_consensus.cc:2258] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:32:58.188704 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 22119
I20260929 16:32:58.206933 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 22271
W20260929 16:32:58.206923 22068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.232.129:41959 (error 108)
I20260929 16:32:58.209255 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.208711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.0.0.1:50454)
0929 16:32:58.208957 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:58.209077 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260929 16:32:58.209609 22068 catalog_manager.cc:4584] TS 88a0a340f85240b3a5581a550267ab0f (127.5.232.129:41959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 227f9324e45942cc8ad71215d02bab67: Network error: Client connection negotiation failed: client connection to 127.5.232.129:41959: connect: Connection refused (error 111)
I20260929 16:32:58.223362 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.208227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.5.232.130:45039)
0929 16:32:58.208436 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:58.208442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:58.208467 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:32:58.208537 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:58.208541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:58.208668 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:58.208784 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:58.210564 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:58.211413 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:58.212755 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:58.213195 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:58.213340 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:58.216272 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:58.216287 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:58.216303 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:58.216344 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:58.219841 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:58.220778 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:58.220791 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:58.220802 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:58.220989 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:58.221705 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:58.221708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:58.221711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:58.222062 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:58.222421 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:58.222723 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 16:32:58.222952 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:32:58.224186 22394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:58.263159 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.262666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.0.0.1:50466)
0929 16:32:58.262902 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:58.263005 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 16:32:58.264276 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.263912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.0.0.1:50474)
0929 16:32:58.264076 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:58.264164 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:32:58.325029 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.324500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.0.0.1:50490)
0929 16:32:58.324753 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:58.324871 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 16:32:58.326179 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.325815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.0.0.1:50502)
0929 16:32:58.325976 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:58.326065 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:32:58.339372 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 22271
I20260929 16:32:58.355425 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 22402
I20260929 16:32:58.369014 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.356196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.5.232.131:38417)
0929 16:32:58.356371 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:32:58.356376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:32:58.356394 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:32:58.356507 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:32:58.356511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:32:58.356591 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:32:58.356680 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:32:58.358165 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:58.358908 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:58.360182 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:32:58.360568 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:32:58.360661 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:32:58.363409 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:32:58.363423 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:32:58.363438 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:32:58.363474 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:32:58.366396 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:58.367178 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:58.367184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:58.367186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:58.367321 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:32:58.367685 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:32:58.367688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:32:58.367690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:32:58.368033 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:32:58.368245 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 16:32:58.368564 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 16:32:58.368735 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:32:58.369764 22525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:58.406821 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.406226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.0.0.1:50518)
0929 16:32:58.406481 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:58.406601 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 16:32:58.408089 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:32:58.407610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:41959 (local address 127.0.0.1:50520)
0929 16:32:58.407844 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 16:32:58.407940 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 16:32:58.459630 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 22402
I20260929 16:32:58.476574 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 22049
I20260929 16:32:58.477882 22112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:32:58.592350 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 22049
I20260929 16:33:18.616402 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42825
--webserver_interface=127.5.232.190
--webserver_port=46425
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:18.940891 22578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:18.941186 22578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:18.941365 22578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:18.948660 22578 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:33:18.948768 22578 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:33:18.948866 22578 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:33:18.948961 22578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:18.949054 22578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:18.949180 22578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:18.949262 22578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:18.949344 22578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:18.959468 22578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=46425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:18.960799 22578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:18.962563 22578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:18.974447 22583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:18.974401 22584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:18.974479 22586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:18.975224 22578 server_base.cc:1034] running on GCE node
I20260929 16:33:18.976022 22578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:18.978284 22578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:18.979528 22578 hybrid_clock.cc:648] HybridClock initialized: now 1790699598979472 us; error 40 us; skew 500 ppm
I20260929 16:33:18.979875 22578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:18 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699598, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:18.986186 22578 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:33:18.987665 22578 webserver.cc:460] Webserver started at http://127.5.232.190:46425/ using document root <none> and password file <none>
I20260929 16:33:18.988397 22578 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:18.988531 22578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:18.994022 22578 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:33:18.996842 22593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:18.998230 22578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:18.998417 22578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "a386273cf011d27e015bb7838c25d23b"
server_key_iv: "720ff0ea643584211674f976f68f2b19"
server_key_version: "encryptionkey@0"
I20260929 16:33:18.999451 22578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:19.021131 22578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:19.025540 22578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:19.025907 22578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:19.045109 22578 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:42825
I20260929 16:33:19.045133 22644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:42825 every 8 connection(s)
I20260929 16:33:19.046720 22578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:19.050012 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 22578
I20260929 16:33:19.051033 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:41959
--local_ip_for_outbound_sockets=127.5.232.129
--tserver_master_addrs=127.5.232.190:42825
--webserver_port=34527
--webserver_interface=127.5.232.129
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:19.055914 22645 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:33:19.061203 22645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Bootstrap starting.
I20260929 16:33:19.082532 22645 log.cc:824] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:19.100045 22645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:19.100723 22645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Bootstrap complete.
I20260929 16:33:19.110617 22645 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:33:19.111558 22645 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7eaf0bd0d94a6daf8b8041e7f10ed8, State: Initialized, Role: FOLLOWER
I20260929 16:33:19.112336 22645 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:33:19.112582 22645 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:19.112708 22645 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:19.112910 22645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:33:19.116974 22645 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:33:19.117594 22645 leader_election.cc:302] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf7eaf0bd0d94a6daf8b8041e7f10ed8; no voters:
I20260929 16:33:19.118336 22645 leader_election.cc:288] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:33:19.118474 22649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:33:19.119771 22649 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [term 3 LEADER]: Becoming Leader. State: Replica: bf7eaf0bd0d94a6daf8b8041e7f10ed8, State: Running, Role: LEADER
I20260929 16:33:19.120446 22649 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } }
I20260929 16:33:19.120730 22645 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:19.123215 22650 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } } }
I20260929 16:33:19.123692 22650 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:19.127939 22651 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf7eaf0bd0d94a6daf8b8041e7f10ed8. Latest consensus state: current_term: 3 leader_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf7eaf0bd0d94a6daf8b8041e7f10ed8" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42825 } } }
I20260929 16:33:19.128455 22651 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:19.129340 22655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:19.138748 22655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:19.140479 22655 catalog_manager.cc:1259] Loaded cluster ID: b26230192ffb438396ce0d22c645caf5
I20260929 16:33:19.140666 22655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:19.146302 22655 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:19.148104 22655 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Loaded TSK: 0
I20260929 16:33:19.148367 22655 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf7eaf0bd0d94a6daf8b8041e7f10ed8: Loaded TSK: 1
F20260929 16:33:19.149618 22655 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f159c3e99ad google::LogMessage::Fail() at ??:0
@ 0x7f159c3ec88a google::LogMessage::SendToLog() at ??:0
@ 0x7f159c3e94de google::LogMessage::Flush() at ??:0
@ 0x7f159c3eaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f15a014d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f15a013c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f15b1e181c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f15b1e17783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f15b1ea32a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f15b1ea30c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f15b1eed5e9 std::function<>::operator()() at ??:0
@ 0x7f15b1e1bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f15b1e19c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f15b1ea1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f15b1ea1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f159f035345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f159f03891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f159f038799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f159f013eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f15a22ed6db start_thread at ??:0
@ 0x7f159b28361f clone at ??:0
W20260929 16:33:19.445051 22647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:19.445408 22647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:19.445585 22647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:19.453022 22647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:19.453188 22647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:19.453372 22647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:33:19.463613 22647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:41959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=34527
--tserver_master_addrs=127.5.232.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:19.464967 22647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:19.466387 22647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:19.479950 22692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:19.479987 22695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:19.480010 22693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:19.481079 22647 server_base.cc:1034] running on GCE node
I20260929 16:33:19.481863 22647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:19.482939 22647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:19.484187 22647 hybrid_clock.cc:648] HybridClock initialized: now 1790699599484133 us; error 40 us; skew 500 ppm
I20260929 16:33:19.484536 22647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:19 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699599, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:19.487784 22647 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:33:19.489499 22647 webserver.cc:460] Webserver started at http://127.5.232.129:34527/ using document root <none> and password file <none>
I20260929 16:33:19.490228 22647 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:19.490388 22647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:19.495944 22647 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:33:19.498946 22702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:19.500442 22647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:33:19.500619 22647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "88a0a340f85240b3a5581a550267ab0f"
format_stamp: "Formatted at 2026-09-29 16:32:33 on dist-test-slave-dssx"
server_key: "90cb1422341932745693163b61e0f2f2"
server_key_iv: "aed754f53b1f5659ec3e733d305cdb83"
server_key_version: "encryptionkey@0"
I20260929 16:33:19.501700 22647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:19.524185 22647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:19.528537 22647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:19.528855 22647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:19.529984 22647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:19.531877 22709 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 16:33:19.537487 22647 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:33:19.537644 22647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 16:33:19.537786 22647 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:33:19.540838 22647 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:33:19.540946 22647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260929 16:33:19.541501 22709 tablet_bootstrap.cc:492] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Bootstrap starting.
I20260929 16:33:19.568212 22709 log.cc:824] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:19.579734 22709 tablet_bootstrap.cc:492] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:33:19.580588 22709 tablet_bootstrap.cc:492] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Bootstrap complete.
I20260929 16:33:19.582319 22709 ts_tablet_manager.cc:1397] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.015s
I20260929 16:33:19.592036 22647 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:41959
I20260929 16:33:19.592057 22816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:41959 every 8 connection(s)
I20260929 16:33:19.593468 22647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:19.592671 22709 raft_consensus.cc:348] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:33:19.593775 22709 raft_consensus.cc:729] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a0a340f85240b3a5581a550267ab0f, State: Initialized, Role: FOLLOWER
I20260929 16:33:19.594645 22709 consensus_queue.cc:260] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:33:19.596498 22709 ts_tablet_manager.cc:1428] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260929 16:33:19.598585 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:19.596404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:34995)
0929 16:33:19.597331 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:19.597667 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":458,"threads_started":1}
W20260929 16:33:19.599220 22817 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:42825 (0 consecutive failures): Network error: Failed to ping master at 127.5.232.190:42825: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
I20260929 16:33:19.600139 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:19.599785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:41807)
0929 16:33:19.599946 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:19.600027 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:33:19.600194 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 22647
I20260929 16:33:19.600879 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:45039
--local_ip_for_outbound_sockets=127.5.232.130
--tserver_master_addrs=127.5.232.190:42825
--webserver_port=39037
--webserver_interface=127.5.232.130
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:19.602401 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:19.601894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:51953)
0929 16:33:19.602095 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:19.602210 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:33:19.602713 22817 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:33:19.942401 22821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:19.942762 22821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:19.942940 22821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:19.950240 22821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:19.950412 22821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:19.950609 22821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:33:19.960456 22821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:45039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=39037
--tserver_master_addrs=127.5.232.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:19.962198 22821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:19.963518 22821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:19.976933 22829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:19.976995 22826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:19.977025 22827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:19.977947 22821 server_base.cc:1034] running on GCE node
I20260929 16:33:19.978664 22821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:19.979655 22821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:19.980937 22821 hybrid_clock.cc:648] HybridClock initialized: now 1790699599980880 us; error 67 us; skew 500 ppm
I20260929 16:33:19.981292 22821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:19 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699599, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:19.984397 22821 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:33:19.985911 22821 webserver.cc:460] Webserver started at http://127.5.232.130:39037/ using document root <none> and password file <none>
I20260929 16:33:19.986615 22821 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:19.986764 22821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:19.992640 22821 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 16:33:19.995784 22836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:19.997138 22821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:33:19.997334 22821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "9f75774c887b49b9820de2f25172ae5c"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "3f1b4c94aa21aa47c4e9d9980c07764d"
server_key_iv: "bc0e02312fdc84051c7907aebd7bdd30"
server_key_version: "encryptionkey@0"
I20260929 16:33:19.998420 22821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:20.028333 22821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:20.032369 22821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:20.032666 22821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:20.033675 22821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:20.035112 22821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:20.035233 22821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:20.035370 22821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:20.035459 22821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:20.068789 22821 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:45039
I20260929 16:33:20.068881 22948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:45039 every 8 connection(s)
I20260929 16:33:20.070358 22821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:20.075421 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:20.073438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:55651)
0929 16:33:20.074168 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:20.074401 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":271,"threads_started":1}
I20260929 16:33:20.075592 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 22821
W20260929 16:33:20.076206 22949 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:42825 (0 consecutive failures): Network error: Failed to ping master at 127.5.232.190:42825: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
I20260929 16:33:20.076398 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:38417
--local_ip_for_outbound_sockets=127.5.232.131
--tserver_master_addrs=127.5.232.190:42825
--webserver_port=41165
--webserver_interface=127.5.232.131
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:33:20.077173 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:20.076854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:37839)
0929 16:33:20.077002 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:20.077070 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:33:20.078061 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:20.077745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:40339)
0929 16:33:20.077883 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:20.077963 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:33:20.078308 22949 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:33:20.407135 22952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:20.407462 22952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:20.407625 22952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:20.414961 22952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:20.415123 22952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:20.415313 22952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:33:20.425052 22952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:38417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=41165
--tserver_master_addrs=127.5.232.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:20.426416 22952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:20.427762 22952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:20.441027 22958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:20.441176 22957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:20.441152 22960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:20.442359 22952 server_base.cc:1034] running on GCE node
I20260929 16:33:20.443025 22952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:20.444046 22952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:20.445271 22952 hybrid_clock.cc:648] HybridClock initialized: now 1790699600445205 us; error 60 us; skew 500 ppm
I20260929 16:33:20.445619 22952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:20 dist-test-slave-dssx krb5kdc[21498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699600, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:20.448776 22952 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:33:20.450343 22952 webserver.cc:460] Webserver started at http://127.5.232.131:41165/ using document root <none> and password file <none>
I20260929 16:33:20.451042 22952 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:20.451191 22952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:20.456564 22952 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:33:20.459321 22967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:20.460680 22952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:33:20.460876 22952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "642f2990e3b84ff7b6c6584fe57df64a"
format_stamp: "Formatted at 2026-09-29 16:32:34 on dist-test-slave-dssx"
server_key: "788e8cb8ff5274699380a7881682620b"
server_key_iv: "ece20ef5d0418be1d9af8bd3c8451b3c"
server_key_version: "encryptionkey@0"
I20260929 16:33:20.461961 22952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:20.493153 22952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:20.498275 22952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:20.498579 22952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:20.499733 22952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:20.501204 22952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:20.501353 22952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:20.501531 22952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:20.501622 22952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:20.534157 22952 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:38417
I20260929 16:33:20.534205 23079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:38417 every 8 connection(s)
I20260929 16:33:20.535681 22952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:20.540514 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 22952
I20260929 16:33:20.540675 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:20.538761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:41233)
0929 16:33:20.539510 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:20.539740 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":295,"threads_started":1}
W20260929 16:33:20.541411 23080 heartbeater.cc:644] Failed to heartbeat to 127.5.232.190:42825 (0 consecutive failures): Network error: Failed to ping master at 127.5.232.190:42825: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
I20260929 16:33:20.542424 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:20.542017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:58727)
0929 16:33:20.542191 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:20.542285 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 16:33:20.543296 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:20.542944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:38925)
0929 16:33:20.543095 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:20.543183 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:33:20.543560 23080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:33:20.604084 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:20.603244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:37223)
0929 16:33:20.603771 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:20.603870 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":239,"threads_started":1}
I20260929 16:33:21.079901 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:21.078909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:36927)
0929 16:33:21.079488 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:21.079582 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":291,"threads_started":1}
I20260929 16:33:21.167534 23086 raft_consensus.cc:482] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:21.167910 23086 raft_consensus.cc:504] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:33:21.169605 23086 leader_election.cc:288] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417), 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039)
Sep 29 16:33:21 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699599, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
Sep 29 16:33:21 dist-test-slave-dssx krb5kdc[21498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699599, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:33:21.193508 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:21.170337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:54403 (local address 127.5.232.130:45039)
0929 16:33:21.171110 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:21.171133 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:21.171200 (+ 67us) server_negotiation.cc:407] Connection header received
0929 16:33:21.173454 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:21.173465 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:21.173862 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:21.174208 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:21.176192 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.177354 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.179519 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.180344 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.180543 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:21.184764 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:21.184775 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:21.184794 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:21.184858 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:21.188876 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:21.190228 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:21.190238 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:21.190245 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:21.190443 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:21.191249 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:21.191254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:21.191258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:21.191883 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:21.192075 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:21.192680 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 16:33:21.193041 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":622,"threads_started":1}
I20260929 16:33:21.193531 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:21.170760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:43583 (local address 127.5.232.131:38417)
0929 16:33:21.171674 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:21.171688 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:21.172088 (+ 400us) server_negotiation.cc:407] Connection header received
0929 16:33:21.173799 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:21.173808 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:21.174141 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:21.174462 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:21.176188 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.177353 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.179197 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.180106 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.180314 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:21.184573 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:21.184584 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:21.184607 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:21.184666 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:21.188876 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:21.190165 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:21.190176 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:21.190184 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:21.190444 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:21.190963 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:21.190968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:21.190971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:21.191683 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:21.191880 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:21.192677 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 16:33:21.193071 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":590,"threads_started":1}
I20260929 16:33:21.193545 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:21.169980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:38417 (local address 127.5.232.129:43583)
0929 16:33:21.171971 (+ 1991us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:21.171988 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:21.172118 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:21.174409 (+ 2291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:21.174414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:21.175237 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:21.175981 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:21.176002 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.177624 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.177629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:21.179048 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:21.179061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.180414 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.180420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:21.180676 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:21.181596 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:21.181613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:21.184218 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:21.189114 (+ 4896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:21.189122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:21.189135 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:21.190001 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:21.190587 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:21.190592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:21.190595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:21.190846 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:21.191840 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:21.191858 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:21.192152 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 16:33:21.192435 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 16:33:21.193071 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1862,"mutex_wait_us":126,"thread_start_us":1682,"threads_started":1}
I20260929 16:33:21.193745 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:21.169929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:45039 (local address 127.5.232.129:54403)
0929 16:33:21.170543 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:21.170592 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:21.171175 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:21.174320 (+ 3145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:21.174334 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:21.175236 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:21.175981 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:21.176001 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.177610 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.177614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:21.179054 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:21.179066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:21.180540 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:21.180547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:21.180756 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:21.181518 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:21.181545 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:21.184478 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:21.189511 (+ 5033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:21.189516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:21.189519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:21.190107 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:21.190957 (+ 850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:21.190961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:21.190963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:21.191082 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:21.192029 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:21.192034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:21.192197 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 16:33:21.192435 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 16:33:21.193234 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":447,"spinlock_wait_cycles":22272,"thread_start_us":282,"threads_started":1}
W20260929 16:33:21.197162 22704 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
W20260929 16:33:21.197527 22706 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
I20260929 16:33:21.197721 22706 leader_election.cc:302] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a0a340f85240b3a5581a550267ab0f; no voters: 642f2990e3b84ff7b6c6584fe57df64a, 9f75774c887b49b9820de2f25172ae5c
I20260929 16:33:21.198154 23086 raft_consensus.cc:2726] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:33:21.544968 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:21.544121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:59431)
0929 16:33:21.544580 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:21.544730 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":215,"threads_started":1}
I20260929 16:33:21.605602 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:21.605037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:43563)
0929 16:33:21.605224 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:21.605321 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:33:22.081611 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:22.080874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:59891)
0929 16:33:22.081293 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:22.081438 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":225,"threads_started":1}
I20260929 16:33:22.546772 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:22.546042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:33641)
0929 16:33:22.546513 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:22.546603 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":250,"threads_started":1}
I20260929 16:33:22.607344 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:22.606621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:50137)
0929 16:33:22.607059 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:22.607146 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":222,"threads_started":1}
I20260929 16:33:22.969972 23095 raft_consensus.cc:482] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:22.970222 23095 raft_consensus.cc:504] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:33:22.970942 23095 leader_election.cc:288] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417), 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039)
W20260929 16:33:22.971912 22704 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
W20260929 16:33:22.972133 22706 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
I20260929 16:33:22.972285 22706 leader_election.cc:302] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a0a340f85240b3a5581a550267ab0f; no voters: 642f2990e3b84ff7b6c6584fe57df64a, 9f75774c887b49b9820de2f25172ae5c
I20260929 16:33:22.972554 23095 raft_consensus.cc:2726] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:33:23.083215 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:23.082505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:53013)
0929 16:33:23.082958 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:23.083043 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":245,"threads_started":1}
I20260929 16:33:23.548472 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:23.547727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:56273)
0929 16:33:23.548208 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:23.548309 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":253,"threads_started":1}
I20260929 16:33:23.609521 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:23.608467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:59407)
0929 16:33:23.609201 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:23.609296 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":477,"threads_started":1}
I20260929 16:33:24.084934 23099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.084162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:52735)
0929 16:33:24.084652 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.084749 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":255,"threads_started":1}
I20260929 16:33:24.550426 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.549640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:36583)
0929 16:33:24.550154 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.550239 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":290,"threads_started":1}
I20260929 16:33:24.611305 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:24.610583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:44003)
0929 16:33:24.611041 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:24.611120 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":215,"threads_started":1}
I20260929 16:33:24.676770 23102 raft_consensus.cc:482] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:24.677037 23102 raft_consensus.cc:504] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:33:24.677814 23102 leader_election.cc:288] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417), 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039)
W20260929 16:33:24.678792 22704 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
W20260929 16:33:24.679023 22706 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
I20260929 16:33:24.679167 22706 leader_election.cc:302] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a0a340f85240b3a5581a550267ab0f; no voters: 642f2990e3b84ff7b6c6584fe57df64a, 9f75774c887b49b9820de2f25172ae5c
I20260929 16:33:24.679437 23102 raft_consensus.cc:2726] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:33:25.086834 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:25.086074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:36401)
0929 16:33:25.086573 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:25.086664 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":261,"threads_started":1}
I20260929 16:33:25.552168 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:25.551419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:35821)
0929 16:33:25.551917 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:25.552007 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":252,"threads_started":1}
I20260929 16:33:25.612933 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:25.612235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:41095)
0929 16:33:25.612660 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:25.612764 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":218,"threads_started":1}
I20260929 16:33:26.088587 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:26.087840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:34577)
0929 16:33:26.088315 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:26.088393 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":267,"threads_started":1}
I20260929 16:33:26.554054 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:26.553228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:33689)
0929 16:33:26.553778 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:26.553870 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":327,"threads_started":1}
I20260929 16:33:26.615074 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:26.614057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:56013)
0929 16:33:26.614670 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:26.614782 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":335,"threads_started":1}
I20260929 16:33:27.090499 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.089638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:59533)
0929 16:33:27.090208 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.090301 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260929 16:33:27.555928 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.555084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:51403)
0929 16:33:27.555638 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.555733 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":261,"threads_started":1}
I20260929 16:33:27.616846 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:27.616113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:58833)
0929 16:33:27.616593 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:27.616686 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":243,"threads_started":1}
I20260929 16:33:27.866072 23112 raft_consensus.cc:482] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:27.866310 23112 raft_consensus.cc:504] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:33:27.866991 23112 leader_election.cc:288] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417), 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039)
W20260929 16:33:27.867903 22704 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
W20260929 16:33:27.868076 22706 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
I20260929 16:33:27.868218 22706 leader_election.cc:302] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a0a340f85240b3a5581a550267ab0f; no voters: 642f2990e3b84ff7b6c6584fe57df64a, 9f75774c887b49b9820de2f25172ae5c
I20260929 16:33:27.868501 23112 raft_consensus.cc:2726] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:33:28.092132 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:28.091409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:40777)
0929 16:33:28.091859 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:28.091955 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":232,"threads_started":1}
I20260929 16:33:28.557860 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:28.557055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:42599)
0929 16:33:28.557579 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:28.557672 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":234,"threads_started":1}
I20260929 16:33:28.618985 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:28.618069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:57973)
0929 16:33:28.618706 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:28.618794 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":318,"threads_started":1}
I20260929 16:33:29.094009 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:29.093091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:57967)
0929 16:33:29.093710 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:29.093817 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":350,"threads_started":1}
I20260929 16:33:29.559648 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:29.558857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:56097)
0929 16:33:29.559398 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:29.559491 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":267,"threads_started":1}
I20260929 16:33:29.620697 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:29.620001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:46271)
0929 16:33:29.620454 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:29.620537 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":235,"threads_started":1}
I20260929 16:33:30.095786 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:30.095013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:55151)
0929 16:33:30.095508 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:30.095599 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
I20260929 16:33:30.561760 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:30.560775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:42685)
0929 16:33:30.561333 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:30.561501 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":339,"threads_started":1}
I20260929 16:33:30.622385 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:30.621657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:44121)
0929 16:33:30.622113 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:30.622198 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260929 16:33:31.097762 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:31.096881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:41335)
0929 16:33:31.097325 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:31.097585 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260929 16:33:31.563449 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:31.562719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:44905)
0929 16:33:31.563197 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:31.563296 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":228,"threads_started":1}
I20260929 16:33:31.624188 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:31.623435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:39829)
0929 16:33:31.623915 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:31.624017 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":248,"threads_started":1}
I20260929 16:33:32.099656 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.098704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:54911)
0929 16:33:32.099271 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.099397 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":346,"threads_started":1}
I20260929 16:33:32.565214 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.564420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:55313)
0929 16:33:32.564900 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.565032 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":259,"threads_started":1}
I20260929 16:33:32.626091 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:32.625263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:39113)
0929 16:33:32.625814 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:32.625911 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":338,"threads_started":1}
I20260929 16:33:33.101508 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.100644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:60303)
0929 16:33:33.101164 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.101261 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":285,"threads_started":1}
I20260929 16:33:33.419083 23129 raft_consensus.cc:482] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:33.419355 23129 raft_consensus.cc:504] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88a0a340f85240b3a5581a550267ab0f" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 41959 } } peers { permanent_uuid: "642f2990e3b84ff7b6c6584fe57df64a" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 38417 } } peers { permanent_uuid: "9f75774c887b49b9820de2f25172ae5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 45039 } }
I20260929 16:33:33.420107 23129 leader_election.cc:288] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417), 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039)
W20260929 16:33:33.421178 22704 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 642f2990e3b84ff7b6c6584fe57df64a (127.5.232.131:38417): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
W20260929 16:33:33.421352 22706 leader_election.cc:341] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f75774c887b49b9820de2f25172ae5c (127.5.232.130:45039): Not found: Tablet not found: 227f9324e45942cc8ad71215d02bab67
I20260929 16:33:33.421523 22706 leader_election.cc:302] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a0a340f85240b3a5581a550267ab0f; no voters: 642f2990e3b84ff7b6c6584fe57df64a, 9f75774c887b49b9820de2f25172ae5c
I20260929 16:33:33.421748 23129 raft_consensus.cc:2726] T 227f9324e45942cc8ad71215d02bab67 P 88a0a340f85240b3a5581a550267ab0f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:33:33.566991 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.566195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:37725)
0929 16:33:33.566713 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.566816 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":254,"threads_started":1}
I20260929 16:33:33.628027 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:33.627123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:46359)
0929 16:33:33.627636 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:33.627753 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260929 16:33:34.103276 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.102484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:36509)
0929 16:33:34.102986 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.103078 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260929 16:33:34.568872 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.567951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:48019)
0929 16:33:34.568501 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.568623 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":300,"threads_started":1}
I20260929 16:33:34.630158 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:34.629322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.129:46271)
0929 16:33:34.629857 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:34.629945 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":317,"threads_started":1}
I20260929 16:33:35.105240 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.104435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.130:48547)
0929 16:33:35.104945 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:35.105055 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":212,"threads_started":1}
I20260929 16:33:35.542382 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 22647
I20260929 16:33:35.557210 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.543462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.5.232.129:41959)
0929 16:33:35.544058 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:35.544065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:35.544094 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:33:35.544219 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:35.544229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:35.544571 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:35.544780 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:35.546365 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.547318 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.548529 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.549101 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.549203 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:35.551825 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:35.551838 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:35.551852 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:35.551888 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:35.554777 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.555451 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.555459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.555461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.555596 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.555907 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.555910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.555912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.556286 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:35.556516 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:35.556733 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 16:33:35.556931 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":302,"threads_started":1}
I20260929 16:33:35.558542 22812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:35.570983 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.570225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42825 (local address 127.5.232.131:55063)
0929 16:33:35.570715 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:35.570814 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.190:42825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":254,"threads_started":1}
I20260929 16:33:35.652069 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 22647
I20260929 16:33:35.667253 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 22821
I20260929 16:33:35.681272 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.668049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.5.232.130:45039)
0929 16:33:35.668535 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:35.668539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:35.668555 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:33:35.668610 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:35.668613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:35.668681 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:35.668804 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:35.670316 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.671082 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.672417 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.672850 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.672993 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:35.675823 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:35.675828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:35.675832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:35.675868 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:35.678516 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.679233 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.679239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.679243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.679394 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.679776 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.679782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.679785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.680076 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:35.680377 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:35.680656 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 16:33:35.680885 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":245,"threads_started":1}
I20260929 16:33:35.682159 22944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:35.773128 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 22821
I20260929 16:33:35.790647 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 22952
I20260929 16:33:35.804365 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:35.791445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.5.232.131:38417)
0929 16:33:35.791969 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:35.791974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:35.791992 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:33:35.792049 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:35.792053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:35.792119 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:35.792213 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:35.793614 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.794408 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.795834 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:35.796265 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:35.796365 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:35.799087 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:35.799094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:35.799097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:35.799129 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:35.801846 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.802507 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.802511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.802513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.802671 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:35.803071 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:35.803076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:35.803078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:35.803329 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:35.803538 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:35.803791 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:33:35.803962 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
I20260929 16:33:35.805066 23075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:35.890743 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 22952
2026-09-29T16:33:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64513 ms)
[----------] 30 tests from SecurityITest (300985 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:33:35 dist-test-slave-dssx krb5kdc[23146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:33:35 dist-test-slave-dssx krb5kdc[23146](info): set up 2 sockets
Sep 29 16:33:35 dist-test-slave-dssx krb5kdc[23146](info): commencing operation
krb5kdc: starting...
W20260929 16:33:37.002209 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:33:37 dist-test-slave-dssx krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:33:37.256976 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40435
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:40435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:37.600397 23159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:37.600718 23159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:37.600813 23159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:37.608282 23159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:37.608413 23159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:37.608461 23159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:37.608525 23159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:37.608572 23159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:37.618757 23159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:40435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:37.620187 23159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:37.621508 23159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:37.633749 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:37.633679 23167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:37.633726 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:37.634466 23159 server_base.cc:1034] running on GCE node
I20260929 16:33:37.635257 23159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:37.636726 23159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:37.638000 23159 hybrid_clock.cc:648] HybridClock initialized: now 1790699617637952 us; error 48 us; skew 500 ppm
I20260929 16:33:37.638384 23159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:37 dist-test-slave-dssx krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699617, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:37.641532 23159 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:33:37.642953 23159 webserver.cc:460] Webserver started at http://127.5.232.190:38221/ using document root <none> and password file <none>
I20260929 16:33:37.643620 23159 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:37.643718 23159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:37.643971 23159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:37.647006 23159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "671a264226d9483d96fbab559a58f0e6"
format_stamp: "Formatted at 2026-09-29 16:33:37 on dist-test-slave-dssx"
server_key: "60d598a740a6b48360c736ead5eea772"
server_key_iv: "7ae19dde2965cf32f3a7e548f04833da"
server_key_version: "encryptionkey@0"
I20260929 16:33:37.647734 23159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "671a264226d9483d96fbab559a58f0e6"
format_stamp: "Formatted at 2026-09-29 16:33:37 on dist-test-slave-dssx"
server_key: "60d598a740a6b48360c736ead5eea772"
server_key_iv: "7ae19dde2965cf32f3a7e548f04833da"
server_key_version: "encryptionkey@0"
I20260929 16:33:37.653020 23159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:33:37.656598 23174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:37.657991 23159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:33:37.658155 23159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "671a264226d9483d96fbab559a58f0e6"
format_stamp: "Formatted at 2026-09-29 16:33:37 on dist-test-slave-dssx"
server_key: "60d598a740a6b48360c736ead5eea772"
server_key_iv: "7ae19dde2965cf32f3a7e548f04833da"
server_key_version: "encryptionkey@0"
I20260929 16:33:37.658332 23159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:37.675097 23159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:37.679227 23159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:37.679466 23159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:37.698177 23159 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:40435
I20260929 16:33:37.698185 23225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:40435 every 8 connection(s)
I20260929 16:33:37.699703 23159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:37.701239 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 23159
I20260929 16:33:37.701496 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:33:37.702136 6050 external_mini_cluster.cc:1428] Setting key 4affb28d6a8c9ea94aed1cc0ffc48d58
I20260929 16:33:37.704061 23226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:33:37 dist-test-slave-dssx krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:37.718199 23226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6: Bootstrap starting.
I20260929 16:33:37.722748 23226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:37.724169 23226 log.cc:824] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:37.727996 23226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6: No bootstrap required, opened a new log
I20260929 16:33:37.728318 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:37.704097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.5.232.190:40435)
0929 16:33:37.704967 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:37.704978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:37.705021 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:33:37.706729 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:37.706752 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:37.707165 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:37.707508 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:37.709442 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:37.710787 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:37.712747 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:37.713494 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:37.713638 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:37.718596 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:37.718608 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:37.718628 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:37.718675 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:37.722817 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:37.723796 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:37.723808 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:37.723821 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:37.724086 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:37.724667 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:37.724674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:37.724679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:37.725528 (+ 849us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:37.725830 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:37.726358 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 16:33:37.726814 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":401,"threads_started":1}
I20260929 16:33:37.736629 23226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671a264226d9483d96fbab559a58f0e6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40435 } }
I20260929 16:33:37.736975 23226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:37.737052 23226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671a264226d9483d96fbab559a58f0e6, State: Initialized, Role: FOLLOWER
I20260929 16:33:37.737650 23226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671a264226d9483d96fbab559a58f0e6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40435 } }
I20260929 16:33:37.737854 23226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:37.737950 23226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:37.738077 23226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:37.740162 23226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671a264226d9483d96fbab559a58f0e6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40435 } }
I20260929 16:33:37.740691 23226 leader_election.cc:302] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 671a264226d9483d96fbab559a58f0e6; no voters:
I20260929 16:33:37.741407 23226 leader_election.cc:288] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:37.741518 23231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:37.742190 23231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [term 1 LEADER]: Becoming Leader. State: Replica: 671a264226d9483d96fbab559a58f0e6, State: Running, Role: LEADER
I20260929 16:33:37.742792 23231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671a264226d9483d96fbab559a58f0e6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40435 } }
I20260929 16:33:37.743707 23226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:37.745664 23232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "671a264226d9483d96fbab559a58f0e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671a264226d9483d96fbab559a58f0e6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40435 } } }
I20260929 16:33:37.746066 23232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:37.747006 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 671a264226d9483d96fbab559a58f0e6. Latest consensus state: current_term: 1 leader_uuid: "671a264226d9483d96fbab559a58f0e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "671a264226d9483d96fbab559a58f0e6" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40435 } } }
I20260929 16:33:37.747334 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:37.749691 23239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:37.754891 23239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:37.763962 23239 catalog_manager.cc:1347] Generated new cluster ID: ec4e09bea029461280cf061327d8cc6e
I20260929 16:33:37.764160 23239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:37.787148 23239 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:37.788545 23239 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:37.803035 23239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6: Generated new TSK 0
I20260929 16:33:37.803960 23239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:33:37.876407 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40435
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:38.221212 23252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:38.221599 23252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:38.221707 23252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:38.229784 23252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:38.229892 23252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:38.230010 23252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:33:38.240648 23252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:38.242026 23252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:38.243307 23252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:38.257030 23258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:38.257066 23260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:38.256996 23257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:38.258280 23252 server_base.cc:1034] running on GCE node
I20260929 16:33:38.258857 23252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:38.259773 23252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:38.260957 23252 hybrid_clock.cc:648] HybridClock initialized: now 1790699618260909 us; error 36 us; skew 500 ppm
I20260929 16:33:38.261297 23252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:38 dist-test-slave-dssx krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699618, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:38.264437 23252 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:33:38.265951 23252 webserver.cc:460] Webserver started at http://127.5.232.129:41701/ using document root <none> and password file <none>
I20260929 16:33:38.266654 23252 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:38.266800 23252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:38.267107 23252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:38.270092 23252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d"
format_stamp: "Formatted at 2026-09-29 16:33:38 on dist-test-slave-dssx"
server_key: "67115447626e8c71ca3e1026f5aaf1a5"
server_key_iv: "2a288173111fc1c28ae5615ff45bea4d"
server_key_version: "encryptionkey@0"
I20260929 16:33:38.270913 23252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d"
format_stamp: "Formatted at 2026-09-29 16:33:38 on dist-test-slave-dssx"
server_key: "67115447626e8c71ca3e1026f5aaf1a5"
server_key_iv: "2a288173111fc1c28ae5615ff45bea4d"
server_key_version: "encryptionkey@0"
I20260929 16:33:38.275950 23252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:33:38.279393 23267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:38.280764 23252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:33:38.280942 23252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d"
format_stamp: "Formatted at 2026-09-29 16:33:38 on dist-test-slave-dssx"
server_key: "67115447626e8c71ca3e1026f5aaf1a5"
server_key_iv: "2a288173111fc1c28ae5615ff45bea4d"
server_key_version: "encryptionkey@0"
I20260929 16:33:38.281160 23252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:38.297284 23252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:38.301513 23252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:38.301822 23252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:38.302934 23252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:38.304369 23252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:38.304443 23252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:38.304533 23252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:38.304579 23252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:38.340376 23252 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:34775
I20260929 16:33:38.340415 23379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:34775 every 8 connection(s)
I20260929 16:33:38.341862 23252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:38.352111 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 23252
I20260929 16:33:38.352339 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:33:38.353101 6050 external_mini_cluster.cc:1428] Setting key 4d3b7e6d4844a65be0143a0cdf80db8f
Sep 29 16:33:38 dist-test-slave-dssx krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699618, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:38.363516 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:38.344553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:34205 (local address 127.5.232.190:40435)
0929 16:33:38.345068 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:38.345073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:38.346055 (+ 982us) server_negotiation.cc:407] Connection header received
0929 16:33:38.348207 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:38.348212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:38.348313 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:38.348437 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:38.350145 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:38.350925 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:38.353199 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:38.353703 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:38.353857 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:38.357514 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:38.357520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:38.357524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:38.357558 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:38.360181 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:38.360986 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:38.360994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:38.360997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:38.361126 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:38.361653 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:38.361660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:38.361666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:38.361943 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:38.362169 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:38.362892 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 16:33:38.363128 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":282,"threads_started":1}
I20260929 16:33:38.364542 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:38.345137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40435 (local address 127.5.232.129:34205)
0929 16:33:38.345895 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:38.345934 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:38.347886 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:38.348692 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:38.348701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:38.349295 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:38.349958 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:38.349976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:38.351115 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:38.351119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:38.353004 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:38.353020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:38.353957 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:38.353966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:38.354240 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:38.354993 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:38.355029 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:38.357247 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:38.360350 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:38.360360 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:38.360380 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:38.360856 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:38.361255 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:38.361260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:38.361264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:38.361539 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:38.362083 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:38.362091 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:38.362492 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 16:33:38.362772 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 16:33:38.363339 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":317,"threads_started":1}
I20260929 16:33:38.366340 23380 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40435
I20260929 16:33:38.366696 23380 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:38.367591 23380 heartbeater.cc:511] Master 127.5.232.190:40435 requested a full tablet report, sending...
I20260929 16:33:38.370246 23191 ts_manager.cc:194] Registered new tserver with Master: ee5c54f1bca34ef88b2040e60f9c0f3d (127.5.232.129:34775)
I20260929 16:33:38.372466 23191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:34205
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:33:38.409258 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40435
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:38.752020 23386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:38.752408 23386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:38.752507 23386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:38.759837 23386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:38.759963 23386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:38.760077 23386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:33:38.770001 23386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:38.771277 23386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:38.772650 23386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:38.785701 23394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:38.785712 23392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:38.785776 23391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:38.787374 23386 server_base.cc:1034] running on GCE node
I20260929 16:33:38.788095 23386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:38.789081 23386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:38.790298 23386 hybrid_clock.cc:648] HybridClock initialized: now 1790699618790224 us; error 69 us; skew 500 ppm
I20260929 16:33:38.790637 23386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:38 dist-test-slave-dssx krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699618, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:38.796950 23386 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:33:38.798485 23386 webserver.cc:460] Webserver started at http://127.5.232.130:44413/ using document root <none> and password file <none>
I20260929 16:33:38.799211 23386 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:38.799357 23386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:38.799687 23386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:38.802938 23386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "897c9f57bd5b4da582c77588486f2e4e"
format_stamp: "Formatted at 2026-09-29 16:33:38 on dist-test-slave-dssx"
server_key: "b47e097a228de5090e1fb50935f09d36"
server_key_iv: "677ad814ea8bec9045a5978447659977"
server_key_version: "encryptionkey@0"
I20260929 16:33:38.803859 23386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "897c9f57bd5b4da582c77588486f2e4e"
format_stamp: "Formatted at 2026-09-29 16:33:38 on dist-test-slave-dssx"
server_key: "b47e097a228de5090e1fb50935f09d36"
server_key_iv: "677ad814ea8bec9045a5978447659977"
server_key_version: "encryptionkey@0"
I20260929 16:33:38.809046 23386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:33:38.812562 23401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:38.814085 23386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:38.814282 23386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "897c9f57bd5b4da582c77588486f2e4e"
format_stamp: "Formatted at 2026-09-29 16:33:38 on dist-test-slave-dssx"
server_key: "b47e097a228de5090e1fb50935f09d36"
server_key_iv: "677ad814ea8bec9045a5978447659977"
server_key_version: "encryptionkey@0"
I20260929 16:33:38.814487 23386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:38.833113 23386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:38.837303 23386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:38.837669 23386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:38.838681 23386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:38.840078 23386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:38.840163 23386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:38.840337 23386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:38.840386 23386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:38.873900 23386 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:35439
I20260929 16:33:38.873946 23513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:35439 every 8 connection(s)
I20260929 16:33:38.875315 23386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:38.884781 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 23386
I20260929 16:33:38.884979 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:33:38.885567 6050 external_mini_cluster.cc:1428] Setting key 9e54235008a7cf2324359f231fdab71c
Sep 29 16:33:38 dist-test-slave-dssx krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699618, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:38.898804 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:38.878054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:51445 (local address 127.5.232.190:40435)
0929 16:33:38.878562 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:38.878567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:38.879473 (+ 906us) server_negotiation.cc:407] Connection header received
0929 16:33:38.881438 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:38.881444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:38.881542 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:38.881670 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:38.883328 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:38.884080 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:38.886611 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:38.887118 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:38.887229 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:38.891741 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:38.891750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:38.891758 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:38.891808 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:38.895476 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:38.896313 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:38.896321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:38.896327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:38.896505 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:38.896939 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:38.896945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:38.896948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:38.897235 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:38.897425 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:38.898151 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 16:33:38.898386 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
I20260929 16:33:38.899778 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:38.878572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40435 (local address 127.5.232.130:51445)
0929 16:33:38.879297 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:38.879341 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:38.881097 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:38.881944 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:38.881954 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:38.882535 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:38.883141 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:38.883159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:38.884270 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:38.884274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:38.886427 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:38.886445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:38.887379 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:38.887387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:38.887697 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:38.888451 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:38.888490 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:38.891448 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:38.895664 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:38.895673 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:38.895688 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:38.896175 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:38.896623 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:38.896626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:38.896629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:38.896809 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:38.897407 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:38.897416 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:38.897780 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 16:33:38.898066 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 16:33:38.898637 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":268,"threads_started":1}
I20260929 16:33:38.901365 23514 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40435
I20260929 16:33:38.901741 23514 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:38.902695 23514 heartbeater.cc:511] Master 127.5.232.190:40435 requested a full tablet report, sending...
I20260929 16:33:38.904433 23191 ts_manager.cc:194] Registered new tserver with Master: 897c9f57bd5b4da582c77588486f2e4e (127.5.232.130:35439)
I20260929 16:33:38.905288 23191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:51445
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:33:38.944319 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40435
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:39.290275 23520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:39.290576 23520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:39.290674 23520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:39.298295 23520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:39.298411 23520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:39.298527 23520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:33:39.308625 23520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:40435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:39.309968 23520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:39.311308 23520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:39.325623 23525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:39.325662 23526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:39.325582 23528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:39.332240 23520 server_base.cc:1034] running on GCE node
I20260929 16:33:39.332947 23520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:39.334085 23520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:39.335316 23520 hybrid_clock.cc:648] HybridClock initialized: now 1790699619335267 us; error 39 us; skew 500 ppm
I20260929 16:33:39.335685 23520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:39 dist-test-slave-dssx krb5kdc[23146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699619, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:39.339000 23520 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:33:39.340700 23520 webserver.cc:460] Webserver started at http://127.5.232.131:40173/ using document root <none> and password file <none>
I20260929 16:33:39.341521 23520 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:39.341672 23520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:39.341992 23520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:39.345156 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "ff866dba7891405a8fe483e186c003e6"
format_stamp: "Formatted at 2026-09-29 16:33:39 on dist-test-slave-dssx"
server_key: "05eee10110723799e843b11ca92beec1"
server_key_iv: "14714972acba3ab77186c5cae6a479bc"
server_key_version: "encryptionkey@0"
I20260929 16:33:39.346014 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "ff866dba7891405a8fe483e186c003e6"
format_stamp: "Formatted at 2026-09-29 16:33:39 on dist-test-slave-dssx"
server_key: "05eee10110723799e843b11ca92beec1"
server_key_iv: "14714972acba3ab77186c5cae6a479bc"
server_key_version: "encryptionkey@0"
I20260929 16:33:39.351557 23520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:33:39.355187 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:39.356575 23520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:33:39.356766 23520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "ff866dba7891405a8fe483e186c003e6"
format_stamp: "Formatted at 2026-09-29 16:33:39 on dist-test-slave-dssx"
server_key: "05eee10110723799e843b11ca92beec1"
server_key_iv: "14714972acba3ab77186c5cae6a479bc"
server_key_version: "encryptionkey@0"
I20260929 16:33:39.356986 23520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:39.376381 23520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:39.376681 23380 heartbeater.cc:503] Master 127.5.232.190:40435 was elected leader, sending a full tablet report...
I20260929 16:33:39.381121 23520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:39.381371 23520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:39.382426 23520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:39.383739 23520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:39.383812 23520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:39.383904 23520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:39.383949 23520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:39.417796 23520 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:36383
I20260929 16:33:39.417845 23647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:36383 every 8 connection(s)
I20260929 16:33:39.419272 23520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:39.419900 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 23520
I20260929 16:33:39.420140 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:33:39.420704 6050 external_mini_cluster.cc:1428] Setting key 2fc4cb2b3a581db3c2699b368301c4eb
Sep 29 16:33:39 dist-test-slave-dssx krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699619, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:39.440420 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.422046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:48441 (local address 127.5.232.190:40435)
0929 16:33:39.422541 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.422548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.423905 (+ 1357us) server_negotiation.cc:407] Connection header received
0929 16:33:39.426012 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.426016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.426101 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.426199 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:39.427801 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.428539 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.430267 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.430674 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.430770 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.434372 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:39.434379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:39.434383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:39.434421 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:39.437663 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:39.438373 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:39.438376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:39.438379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:39.438491 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:39.438879 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:39.438882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:39.438884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:39.439158 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:39.439274 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.439962 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.440134 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
I20260929 16:33:39.441497 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.422822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:40435 (local address 127.5.232.131:48441)
0929 16:33:39.423737 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.423803 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:39.425732 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:39.426428 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:39.426437 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:39.426998 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:39.427635 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.427654 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.428691 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.428695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.430117 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.430130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.430880 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.430885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.431112 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.431875 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:39.431902 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:39.434128 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:39.437804 (+ 3676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:39.437811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:39.437827 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:39.438263 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:39.438594 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:39.438598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:39.438600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:39.438780 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:39.439261 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:39.439267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:39.439620 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 16:33:39.439869 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 16:33:39.440263 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":313,"threads_started":1}
I20260929 16:33:39.442945 23648 heartbeater.cc:347] Connected to a master server at 127.5.232.190:40435
I20260929 16:33:39.443222 23648 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:39.443986 23648 heartbeater.cc:511] Master 127.5.232.190:40435 requested a full tablet report, sending...
I20260929 16:33:39.445514 23191 ts_manager.cc:194] Registered new tserver with Master: ff866dba7891405a8fe483e186c003e6 (127.5.232.131:36383)
I20260929 16:33:39.446460 23191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:48441
I20260929 16:33:39.449466 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:39.467249 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.454078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.5.232.190:40435)
0929 16:33:39.454277 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.454284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.454698 (+ 414us) server_negotiation.cc:407] Connection header received
0929 16:33:39.454908 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.454912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.454988 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.455092 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:39.456620 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.457350 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.458753 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.459175 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.459311 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.461810 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:39.461815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:39.461819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:39.461850 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:39.464412 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:39.465106 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:39.465123 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:39.465131 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:39.465251 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:39.465677 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:39.465684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:39.465689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:39.466033 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:39.466305 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.466665 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.466899 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:33:39.471349 23191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:39.479056 23191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:39.517417 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.492267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:36383 (local address 127.0.0.1:50084)
0929 16:33:39.502877 (+ 10610us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.502905 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:39.503071 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:39.510126 (+ 7055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:39.510131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:39.510158 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:39.510727 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.510740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.512672 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.512678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.514749 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.514770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.516362 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.516366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.516499 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.516534 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:33:39.516685 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 16:33:39.516883 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10423,"mutex_wait_us":150,"thread_start_us":8692,"threads_started":1}
I20260929 16:33:39.518523 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.492141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34775 (local address 127.0.0.1:58440)
0929 16:33:39.492860 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.492903 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:39.493092 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:39.509867 (+ 16775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:39.509876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:39.509909 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:39.510415 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.510429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.512556 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.512562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.515398 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.515413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.517322 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.517332 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.517743 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.517779 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:33:39.517854 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 16:33:39.517986 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":107,"thread_start_us":803,"threads_started":1}
I20260929 16:33:39.518610 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.492329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.5.232.131:36383)
0929 16:33:39.509502 (+ 17173us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.509513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.509547 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:33:39.509655 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.509663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.509904 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.510116 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:39.511017 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.512433 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.514924 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.516176 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.516314 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.516561 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.516742 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.518186 (+ 1444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16945,"thread_start_us":16814,"threads_started":1}
I20260929 16:33:39.519495 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.492767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.5.232.129:34775)
0929 16:33:39.509049 (+ 16282us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.509062 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.509098 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:33:39.509228 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.509239 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.509586 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.510449 (+ 863us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:39.510740 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.512404 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.515574 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.517010 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.518621 (+ 1611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.518879 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.518947 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.519128 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16073,"thread_start_us":15905,"threads_started":1}
I20260929 16:33:39.520491 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.494909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.5.232.130:35439)
0929 16:33:39.509887 (+ 14978us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.509894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.509919 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:33:39.509997 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.510002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.510260 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.510479 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:39.511294 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.512763 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.516671 (+ 3908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.518153 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.518293 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.518498 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.520025 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.520194 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14812,"thread_start_us":14682,"threads_started":1}
I20260929 16:33:39.520496 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.494318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:35439 (local address 127.0.0.1:44384)
0929 16:33:39.508617 (+ 14299us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.508646 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:39.508818 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:39.510627 (+ 1809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:39.510631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:39.510662 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:39.511091 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.511103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.512912 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.512919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.515413 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.515423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.518382 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.518389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.518575 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.519849 (+ 1274us) client_negotiation.cc:769] Sending connection context
0929 16:33:39.519978 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:33:39.520115 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14110,"mutex_wait_us":40,"thread_start_us":13918,"threads_started":1}
I20260929 16:33:39.522941 23583 tablet_service.cc:1467] Processing CreateTablet for tablet acef428e37ec4129b3727141aaf09af5 (DEFAULT_TABLE table=test-table [id=68045494a0444d6d9a86a75e9457c69c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:39.522941 23315 tablet_service.cc:1467] Processing CreateTablet for tablet acef428e37ec4129b3727141aaf09af5 (DEFAULT_TABLE table=test-table [id=68045494a0444d6d9a86a75e9457c69c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:39.524523 23315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acef428e37ec4129b3727141aaf09af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:39.524523 23583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acef428e37ec4129b3727141aaf09af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:39.527846 23449 tablet_service.cc:1467] Processing CreateTablet for tablet acef428e37ec4129b3727141aaf09af5 (DEFAULT_TABLE table=test-table [id=68045494a0444d6d9a86a75e9457c69c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:39.529110 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acef428e37ec4129b3727141aaf09af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:39.539803 23667 tablet_bootstrap.cc:492] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: Bootstrap starting.
I20260929 16:33:39.540627 23668 tablet_bootstrap.cc:492] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: Bootstrap starting.
I20260929 16:33:39.542089 23669 tablet_bootstrap.cc:492] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: Bootstrap starting.
I20260929 16:33:39.544654 23668 tablet_bootstrap.cc:654] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:39.545082 23667 tablet_bootstrap.cc:654] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:39.545596 23669 tablet_bootstrap.cc:654] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:39.546118 23668 log.cc:824] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:39.546509 23667 log.cc:824] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:39.546818 23669 log.cc:824] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:39.549539 23668 tablet_bootstrap.cc:492] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: No bootstrap required, opened a new log
I20260929 16:33:39.549794 23667 tablet_bootstrap.cc:492] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: No bootstrap required, opened a new log
I20260929 16:33:39.549840 23668 ts_tablet_manager.cc:1397] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:33:39.549971 23669 tablet_bootstrap.cc:492] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: No bootstrap required, opened a new log
I20260929 16:33:39.550079 23667 ts_tablet_manager.cc:1397] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 16:33:39.550215 23669 ts_tablet_manager.cc:1397] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 16:33:39.557711 23668 raft_consensus.cc:348] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } }
I20260929 16:33:39.558238 23668 raft_consensus.cc:374] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:39.558338 23668 raft_consensus.cc:729] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5c54f1bca34ef88b2040e60f9c0f3d, State: Initialized, Role: FOLLOWER
I20260929 16:33:39.559145 23668 consensus_queue.cc:260] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } }
I20260929 16:33:39.561303 23668 ts_tablet_manager.cc:1428] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260929 16:33:39.561204 23667 raft_consensus.cc:348] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } }
I20260929 16:33:39.561195 23669 raft_consensus.cc:348] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } }
I20260929 16:33:39.561724 23669 raft_consensus.cc:374] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:39.561715 23667 raft_consensus.cc:374] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:39.561865 23667 raft_consensus.cc:729] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff866dba7891405a8fe483e186c003e6, State: Initialized, Role: FOLLOWER
I20260929 16:33:39.561900 23669 raft_consensus.cc:729] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897c9f57bd5b4da582c77588486f2e4e, State: Initialized, Role: FOLLOWER
I20260929 16:33:39.562520 23667 consensus_queue.cc:260] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } }
I20260929 16:33:39.562517 23669 consensus_queue.cc:260] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } }
I20260929 16:33:39.564157 23648 heartbeater.cc:503] Master 127.5.232.190:40435 was elected leader, sending a full tablet report...
I20260929 16:33:39.564411 23669 ts_tablet_manager.cc:1428] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 16:33:39.564736 23667 ts_tablet_manager.cc:1428] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 16:33:39.565049 23514 heartbeater.cc:503] Master 127.5.232.190:40435 was elected leader, sending a full tablet report...
W20260929 16:33:39.594523 23381 tablet.cc:2369] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:33:39.627506 23515 tablet.cc:2369] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:33:39.635854 23674 raft_consensus.cc:482] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:39.636222 23674 raft_consensus.cc:504] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } }
I20260929 16:33:39.638046 23674 leader_election.cc:288] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5c54f1bca34ef88b2040e60f9c0f3d (127.5.232.129:34775), 897c9f57bd5b4da582c77588486f2e4e (127.5.232.130:35439)
I20260929 16:33:39.645064 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.638511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:33361 (local address 127.5.232.129:34775)
0929 16:33:39.638736 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.638743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.638769 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:33:39.638858 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.638862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.638943 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.639083 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:39.639725 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.640703 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.642994 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.643989 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.644241 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.644323 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.644687 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.644835 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:33:39.645080 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.638338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34775 (local address 127.5.232.131:33361)
0929 16:33:39.638554 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.638576 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:39.638707 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:39.639297 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:39.639302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:39.639329 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:39.639561 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.639574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.640842 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.640846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.642818 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.642833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.644301 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.644306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.644495 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.644545 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 16:33:39.644631 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:33:39.644755 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":1}
I20260929 16:33:39.645764 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.639132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:32787 (local address 127.5.232.130:35439)
0929 16:33:39.639339 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.639343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.639584 (+ 241us) server_negotiation.cc:407] Connection header received
0929 16:33:39.639757 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.639762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.639842 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.639934 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:39.640469 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.641263 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.643530 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.644672 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.644824 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.644888 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.645293 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.645479 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:33:39.646332 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.638744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:35439 (local address 127.5.232.131:32787)
0929 16:33:39.639467 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:39.639488 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:39.639614 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:39.639982 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:39.639986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:39.640008 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:39.640317 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.640325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.641451 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.641456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.643318 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:39.643329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.644877 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.644884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:39.645077 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.645122 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:33:39.645947 (+ 825us) client_negotiation.cc:240] Negotiation successful
0929 16:33:39.646061 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":362,"threads_started":1}
I20260929 16:33:39.646468 23335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acef428e37ec4129b3727141aaf09af5" candidate_uuid: "ff866dba7891405a8fe483e186c003e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" is_pre_election: true
I20260929 16:33:39.647171 23335 raft_consensus.cc:2454] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff866dba7891405a8fe483e186c003e6 in term 0.
I20260929 16:33:39.647408 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acef428e37ec4129b3727141aaf09af5" candidate_uuid: "ff866dba7891405a8fe483e186c003e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897c9f57bd5b4da582c77588486f2e4e" is_pre_election: true
I20260929 16:33:39.648015 23469 raft_consensus.cc:2454] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff866dba7891405a8fe483e186c003e6 in term 0.
I20260929 16:33:39.647989 23538 leader_election.cc:302] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee5c54f1bca34ef88b2040e60f9c0f3d, ff866dba7891405a8fe483e186c003e6; no voters:
I20260929 16:33:39.648447 23674 raft_consensus.cc:2781] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:39.648623 23674 raft_consensus.cc:482] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:39.648732 23674 raft_consensus.cc:3037] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:39.651202 23674 raft_consensus.cc:504] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } }
I20260929 16:33:39.651904 23674 leader_election.cc:288] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [CANDIDATE]: Term 1 election: Requested vote from peers ee5c54f1bca34ef88b2040e60f9c0f3d (127.5.232.129:34775), 897c9f57bd5b4da582c77588486f2e4e (127.5.232.130:35439)
I20260929 16:33:39.652489 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acef428e37ec4129b3727141aaf09af5" candidate_uuid: "ff866dba7891405a8fe483e186c003e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "897c9f57bd5b4da582c77588486f2e4e"
I20260929 16:33:39.652520 23335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acef428e37ec4129b3727141aaf09af5" candidate_uuid: "ff866dba7891405a8fe483e186c003e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d"
I20260929 16:33:39.652705 23469 raft_consensus.cc:3037] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:39.652729 23335 raft_consensus.cc:3037] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:39.655153 23335 raft_consensus.cc:2454] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff866dba7891405a8fe483e186c003e6 in term 1.
I20260929 16:33:39.655227 23469 raft_consensus.cc:2454] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff866dba7891405a8fe483e186c003e6 in term 1.
I20260929 16:33:39.655648 23538 leader_election.cc:302] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee5c54f1bca34ef88b2040e60f9c0f3d, ff866dba7891405a8fe483e186c003e6; no voters:
I20260929 16:33:39.655964 23674 raft_consensus.cc:2781] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:39.656414 23674 raft_consensus.cc:686] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 1 LEADER]: Becoming Leader. State: Replica: ff866dba7891405a8fe483e186c003e6, State: Running, Role: LEADER
I20260929 16:33:39.656908 23674 consensus_queue.cc:237] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } }
I20260929 16:33:39.661949 23189 catalog_manager.cc:5495] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff866dba7891405a8fe483e186c003e6 (127.5.232.131). New cstate: current_term: 1 leader_uuid: "ff866dba7891405a8fe483e186c003e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff866dba7891405a8fe483e186c003e6" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 } health_report { overall_health: UNKNOWN } } }
W20260929 16:33:39.671132 23649 tablet.cc:2369] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:33:39.699110 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.693602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.5.232.131:36383)
0929 16:33:39.693833 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.693840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.693871 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:33:39.694086 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.694090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.694178 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.694307 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:39.694636 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.695353 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.696805 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.697215 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.697311 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.697948 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:39.698487 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:39.698689 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.698739 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.698824 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:33:39.708151 23335 raft_consensus.cc:1261] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [term 1 FOLLOWER]: Refusing update from remote peer ff866dba7891405a8fe483e186c003e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:39.708340 23469 raft_consensus.cc:1261] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [term 1 FOLLOWER]: Refusing update from remote peer ff866dba7891405a8fe483e186c003e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:39.709364 23674 consensus_queue.cc:1035] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5c54f1bca34ef88b2040e60f9c0f3d" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:39.709743 23677 consensus_queue.cc:1035] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "897c9f57bd5b4da582c77588486f2e4e" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 35439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:39.729830 23682 mvcc.cc:205] Tried to move back new op lower bound from 7334705642312978432 to 7334705642119090176. Current Snapshot: MvccSnapshot[applied={T|T < 7334705642312978432}]
I20260929 16:33:39.741021 23189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:39.741634 23189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:39.745467 23189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 671a264226d9483d96fbab559a58f0e6: Sending DeleteTablet for 3 replicas of tablet acef428e37ec4129b3727141aaf09af5
I20260929 16:33:39.746726 23583 tablet_service.cc:1514] Processing DeleteTablet for tablet acef428e37ec4129b3727141aaf09af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:39 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:50084
I20260929 16:33:39.746894 23449 tablet_service.cc:1514] Processing DeleteTablet for tablet acef428e37ec4129b3727141aaf09af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:39 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:44384
I20260929 16:33:39.746894 23315 tablet_service.cc:1514] Processing DeleteTablet for tablet acef428e37ec4129b3727141aaf09af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:39 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:58440
I20260929 16:33:39.749655 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 23252
I20260929 16:33:39.754057 23691 tablet_replica.cc:330] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: stopping tablet replica
I20260929 16:33:39.754673 23691 raft_consensus.cc:2229] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:39.755336 23691 raft_consensus.cc:2258] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:39.756227 23689 tablet_replica.cc:330] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: stopping tablet replica
I20260929 16:33:39.756619 23689 raft_consensus.cc:2229] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:33:39.757169 23689 raft_consensus.cc:2258] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:39.757871 23691 ts_tablet_manager.cc:1905] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:39.758673 23690 tablet_replica.cc:330] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: stopping tablet replica
I20260929 16:33:39.759025 23690 raft_consensus.cc:2229] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:39.759464 23690 raft_consensus.cc:2258] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:39.762190 23690 ts_tablet_manager.cc:1905] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:39.763280 23689 ts_tablet_manager.cc:1905] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:33:39 dist-test-slave-dssx krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:33:39.769706 23690 ts_tablet_manager.cc:1918] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:39.769870 23690 log.cc:1196] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/acef428e37ec4129b3727141aaf09af5
I20260929 16:33:39.770422 23690 ts_tablet_manager.cc:1939] T acef428e37ec4129b3727141aaf09af5 P ee5c54f1bca34ef88b2040e60f9c0f3d: Deleting consensus metadata
I20260929 16:33:39.770778 23689 ts_tablet_manager.cc:1918] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:39.770947 23689 log.cc:1196] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/acef428e37ec4129b3727141aaf09af5
I20260929 16:33:39.771471 23689 ts_tablet_manager.cc:1939] T acef428e37ec4129b3727141aaf09af5 P ff866dba7891405a8fe483e186c003e6: Deleting consensus metadata
I20260929 16:33:39.773360 23176 catalog_manager.cc:4841] TS ff866dba7891405a8fe483e186c003e6 (127.5.232.131:36383): tablet acef428e37ec4129b3727141aaf09af5 (table test-table [id=68045494a0444d6d9a86a75e9457c69c]) successfully deleted
I20260929 16:33:39.773339 23177 catalog_manager.cc:4841] TS ee5c54f1bca34ef88b2040e60f9c0f3d (127.5.232.129:34775): tablet acef428e37ec4129b3727141aaf09af5 (table test-table [id=68045494a0444d6d9a86a75e9457c69c]) successfully deleted
I20260929 16:33:39.776566 23691 ts_tablet_manager.cc:1918] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:39.776772 23691 log.cc:1196] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/acef428e37ec4129b3727141aaf09af5
I20260929 16:33:39.777652 23691 ts_tablet_manager.cc:1939] T acef428e37ec4129b3727141aaf09af5 P 897c9f57bd5b4da582c77588486f2e4e: Deleting consensus metadata
I20260929 16:33:39.778412 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.751593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.5.232.129:34775)
0929 16:33:39.751844 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.751853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.757957 (+ 6104us) server_negotiation.cc:407] Connection header received
0929 16:33:39.758201 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.758215 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.758321 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.758441 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:39.760074 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.761294 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.763668 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.764368 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.764474 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.770539 (+ 6065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:39.770556 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:39.770575 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:39.770634 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:39.774791 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:39.775855 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:39.775865 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:39.775869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:39.776039 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:39.776542 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:39.776547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:39.776551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:39.777061 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:39.777367 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.777781 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.778018 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:33:39.779152 23375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:39.779624 23177 catalog_manager.cc:4841] TS 897c9f57bd5b4da582c77588486f2e4e (127.5.232.130:35439): tablet acef428e37ec4129b3727141aaf09af5 (table test-table [id=68045494a0444d6d9a86a75e9457c69c]) successfully deleted
I20260929 16:33:39.877959 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 23252
I20260929 16:33:39.897234 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 23386
Sep 29 16:33:39 dist-test-slave-dssx krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:33:39.914836 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:39.898063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.5.232.130:35439)
0929 16:33:39.898248 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:39.898254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:39.898396 (+ 142us) server_negotiation.cc:407] Connection header received
0929 16:33:39.898610 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:39.898616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:39.898694 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:39.898805 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:39.900050 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.900862 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.902583 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:39.903001 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:39.903155 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:39.908436 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:39.908452 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:39.908468 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:39.908511 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:39.911756 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:39.912550 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:39.912559 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:39.912562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:39.912690 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:39.913086 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:39.913092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:39.913095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:39.913634 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:39.913933 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:39.914254 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 16:33:39.914477 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:33:39.915589 23509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:40.006372 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 23386
I20260929 16:33:40.029852 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 23520
Sep 29 16:33:40 dist-test-slave-dssx krb5kdc[23146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:33:40.047572 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:40.030874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50092 (local address 127.5.232.131:36383)
0929 16:33:40.031093 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:40.031104 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:40.031120 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:33:40.031178 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:40.031184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:40.031245 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:40.031371 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:40.032827 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:40.033841 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:40.035128 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:40.035606 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:40.035784 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:40.040687 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:40.040701 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:40.040717 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:40.040760 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:40.044048 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:40.045148 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:40.045160 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:40.045163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:40.045354 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:40.045845 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:40.045851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:40.045854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:40.046238 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:40.046573 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:40.046888 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:33:40.047113 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:33:40.048424 23643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:40.172191 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 23520
I20260929 16:33:40.191951 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 23159
I20260929 16:33:40.193092 23221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:40.306720 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 23159
2026-09-29T16:33:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4419 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T16:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:40Z Disabled control of system clock
I20260929 16:33:40.386862 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46075
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:46075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:33:40.733639 23700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:40.733961 23700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:40.734133 23700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:40.741564 23700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:40.741684 23700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:40.741777 23700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:40.741863 23700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:40.741925 23700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:40.751475 23700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:46075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:40.752880 23700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:40.754840 23700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:40.767051 23705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:40.767221 23708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:40.767221 23706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:40.768024 23700 server_base.cc:1034] running on GCE node
I20260929 16:33:40.768857 23700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:40.770493 23700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:40.771749 23700 hybrid_clock.cc:648] HybridClock initialized: now 1790699620771699 us; error 65 us; skew 500 ppm
I20260929 16:33:40.772122 23700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:40.774494 23700 webserver.cc:460] Webserver started at http://127.5.232.190:46623/ using document root <none> and password file <none>
I20260929 16:33:40.775245 23700 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:40.775382 23700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:40.775717 23700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:40.778823 23700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "a2ee070bbe374164a8d967b2405568ec"
format_stamp: "Formatted at 2026-09-29 16:33:40 on dist-test-slave-dssx"
server_key: "ddce06591feaaf59d3d4824956f7cf18"
server_key_iv: "a00dd0362a4324f0b32ada17cc230724"
server_key_version: "encryptionkey@0"
I20260929 16:33:40.779680 23700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "a2ee070bbe374164a8d967b2405568ec"
format_stamp: "Formatted at 2026-09-29 16:33:40 on dist-test-slave-dssx"
server_key: "ddce06591feaaf59d3d4824956f7cf18"
server_key_iv: "a00dd0362a4324f0b32ada17cc230724"
server_key_version: "encryptionkey@0"
I20260929 16:33:40.785281 23700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:33:40.789014 23714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:40.790608 23700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:33:40.790828 23700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "a2ee070bbe374164a8d967b2405568ec"
format_stamp: "Formatted at 2026-09-29 16:33:40 on dist-test-slave-dssx"
server_key: "ddce06591feaaf59d3d4824956f7cf18"
server_key_iv: "a00dd0362a4324f0b32ada17cc230724"
server_key_version: "encryptionkey@0"
I20260929 16:33:40.791083 23700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:40.824661 23700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:40.825766 23700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:40.826086 23700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:40.845264 23700 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:46075
I20260929 16:33:40.845296 23765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:46075 every 8 connection(s)
I20260929 16:33:40.846735 23700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:40.850721 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:40.855978 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 23700
I20260929 16:33:40.856179 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:33:40.856617 6050 external_mini_cluster.cc:1428] Setting key f7e42c7335c08573f9fea8637cdde532
I20260929 16:33:40.863097 23766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec: Bootstrap starting.
I20260929 16:33:40.867252 23766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:40.868542 23766 log.cc:824] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:40.872181 23766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec: No bootstrap required, opened a new log
I20260929 16:33:40.873153 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:40.858726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.5.232.190:46075)
0929 16:33:40.859992 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:40.860007 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:40.860068 (+ 61us) server_negotiation.cc:407] Connection header received
0929 16:33:40.862611 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:40.862642 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:40.863303 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:40.863795 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:40.864367 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:40.866447 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:40.867996 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:40.869164 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:40.869460 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:40.870666 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:40.870673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:40.870688 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:40.871249 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:40.871334 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:40.871357 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:40.871451 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:40.871721 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:33:40.872092 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":442,"threads_started":1}
I20260929 16:33:40.880733 23766 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ee070bbe374164a8d967b2405568ec" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46075 } }
I20260929 16:33:40.881173 23766 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:40.881291 23766 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ee070bbe374164a8d967b2405568ec, State: Initialized, Role: FOLLOWER
I20260929 16:33:40.881956 23766 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ee070bbe374164a8d967b2405568ec" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46075 } }
I20260929 16:33:40.882229 23766 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:40.882380 23766 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:40.882572 23766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:40.884840 23766 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ee070bbe374164a8d967b2405568ec" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46075 } }
I20260929 16:33:40.885545 23766 leader_election.cc:302] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2ee070bbe374164a8d967b2405568ec; no voters:
I20260929 16:33:40.886307 23766 leader_election.cc:288] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:40.886480 23771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:40.887728 23771 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [term 1 LEADER]: Becoming Leader. State: Replica: a2ee070bbe374164a8d967b2405568ec, State: Running, Role: LEADER
I20260929 16:33:40.888271 23771 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ee070bbe374164a8d967b2405568ec" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46075 } }
I20260929 16:33:40.888547 23766 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:40.891554 23773 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ee070bbe374164a8d967b2405568ec. Latest consensus state: current_term: 1 leader_uuid: "a2ee070bbe374164a8d967b2405568ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ee070bbe374164a8d967b2405568ec" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46075 } } }
I20260929 16:33:40.892117 23773 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:40.891592 23772 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2ee070bbe374164a8d967b2405568ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ee070bbe374164a8d967b2405568ec" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46075 } } }
I20260929 16:33:40.892596 23772 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:40.894770 23780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:40.899580 23780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:40.909247 23780 catalog_manager.cc:1347] Generated new cluster ID: 0dae7cf7f76646e1801eabf7b8c014d6
I20260929 16:33:40.909397 23780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:40.943447 23780 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:40.944859 23780 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:40.951957 23780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec: Generated new TSK 0
I20260929 16:33:40.952910 23780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:33:40.967447 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46075
--builtin_ntp_servers=127.5.232.148:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:33:41.322238 23790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:41.322559 23790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:41.322664 23790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:41.330682 23790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:41.330884 23790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:41.331066 23790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:33:41.342500 23790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:41.343823 23790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:41.345733 23790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:41.359462 23798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:41.359407 23796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:41.359498 23795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:41.360312 23790 server_base.cc:1034] running on GCE node
I20260929 16:33:41.361021 23790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:41.362234 23790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:41.363472 23790 hybrid_clock.cc:648] HybridClock initialized: now 1790699621363398 us; error 68 us; skew 500 ppm
I20260929 16:33:41.363783 23790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:41.366192 23790 webserver.cc:460] Webserver started at http://127.5.232.129:36745/ using document root <none> and password file <none>
I20260929 16:33:41.366904 23790 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:41.366993 23790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:41.367274 23790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:41.370350 23790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "1f915f27d1774200baa00b8f88e47db6"
format_stamp: "Formatted at 2026-09-29 16:33:41 on dist-test-slave-dssx"
server_key: "3fafca00bcd77e0c3c4785d0b81f5289"
server_key_iv: "27c6f57b96c202365381752e1cdec654"
server_key_version: "encryptionkey@0"
I20260929 16:33:41.371140 23790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "1f915f27d1774200baa00b8f88e47db6"
format_stamp: "Formatted at 2026-09-29 16:33:41 on dist-test-slave-dssx"
server_key: "3fafca00bcd77e0c3c4785d0b81f5289"
server_key_iv: "27c6f57b96c202365381752e1cdec654"
server_key_version: "encryptionkey@0"
I20260929 16:33:41.376535 23790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:33:41.380232 23804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:41.381731 23790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:41.381916 23790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "1f915f27d1774200baa00b8f88e47db6"
format_stamp: "Formatted at 2026-09-29 16:33:41 on dist-test-slave-dssx"
server_key: "3fafca00bcd77e0c3c4785d0b81f5289"
server_key_iv: "27c6f57b96c202365381752e1cdec654"
server_key_version: "encryptionkey@0"
I20260929 16:33:41.382160 23790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:41.401434 23790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:41.402328 23790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:41.402650 23790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:41.403661 23790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:41.405211 23790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:41.405341 23790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:41.405525 23790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:41.405614 23790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:41.439826 23790 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:39195
I20260929 16:33:41.439888 23916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:39195 every 8 connection(s)
I20260929 16:33:41.441234 23790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:41.443189 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 23790
I20260929 16:33:41.443374 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:33:41.443902 6050 external_mini_cluster.cc:1428] Setting key 1585e02a96fd5426166daffa923578a3
I20260929 16:33:41.447075 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46075
--builtin_ntp_servers=127.5.232.148:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:33:41.456907 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:41.444037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:58117 (local address 127.5.232.190:46075)
0929 16:33:41.444574 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:41.444581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:41.446089 (+ 1508us) server_negotiation.cc:407] Connection header received
0929 16:33:41.448178 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:41.448184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:41.448275 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:41.448419 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:41.449741 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:41.450943 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:41.453338 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:41.454088 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:41.454256 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:41.456094 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:41.456099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:41.456102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:41.456124 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:41.456165 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:41.456195 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:41.456270 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:41.456538 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:33:41.456624 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":258,"threads_started":1}
I20260929 16:33:41.458520 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:41.444651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46075 (local address 127.5.232.129:58117)
0929 16:33:41.445917 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:41.445975 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:41.447886 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:41.448679 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:41.448687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:41.448796 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:41.449540 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:41.449567 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:41.451128 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:41.451136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:41.453152 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:41.453172 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:41.454555 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:41.454564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:41.454854 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:41.455699 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:33:41.455704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:41.455766 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:33:41.455886 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:41.456298 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:41.456312 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:41.456325 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 16:33:41.456526 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 16:33:41.456997 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":365,"threads_started":1}
I20260929 16:33:41.460233 23917 heartbeater.cc:347] Connected to a master server at 127.5.232.190:46075
I20260929 16:33:41.460563 23917 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:41.461465 23917 heartbeater.cc:511] Master 127.5.232.190:46075 requested a full tablet report, sending...
I20260929 16:33:41.463626 23731 ts_manager.cc:194] Registered new tserver with Master: 1f915f27d1774200baa00b8f88e47db6 (127.5.232.129:39195)
I20260929 16:33:41.465129 23731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.129:58117
W20260929 16:33:41.795871 23921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:41.796216 23921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:41.796355 23921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:41.803591 23921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:41.803705 23921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:41.803813 23921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:33:41.813237 23921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:41.814675 23921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:41.816459 23921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:41.829952 23926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:41.829914 23927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:41.830502 23929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:41.830969 23921 server_base.cc:1034] running on GCE node
I20260929 16:33:41.831629 23921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:41.832631 23921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:41.833866 23921 hybrid_clock.cc:648] HybridClock initialized: now 1790699621833826 us; error 62 us; skew 500 ppm
I20260929 16:33:41.834213 23921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:41.836480 23921 webserver.cc:460] Webserver started at http://127.5.232.130:36857/ using document root <none> and password file <none>
I20260929 16:33:41.837211 23921 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:41.837345 23921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:41.837697 23921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:41.840621 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "4c85203c2fe84d6da619d7584274f9e9"
format_stamp: "Formatted at 2026-09-29 16:33:41 on dist-test-slave-dssx"
server_key: "ac3620738c369dfc9e791c9850bc2dd4"
server_key_iv: "257cf63d7ef6f10ffd2d1d1f54383818"
server_key_version: "encryptionkey@0"
I20260929 16:33:41.841512 23921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "4c85203c2fe84d6da619d7584274f9e9"
format_stamp: "Formatted at 2026-09-29 16:33:41 on dist-test-slave-dssx"
server_key: "ac3620738c369dfc9e791c9850bc2dd4"
server_key_iv: "257cf63d7ef6f10ffd2d1d1f54383818"
server_key_version: "encryptionkey@0"
I20260929 16:33:41.846884 23921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:33:41.850626 23935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:41.852101 23921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:33:41.852300 23921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "4c85203c2fe84d6da619d7584274f9e9"
format_stamp: "Formatted at 2026-09-29 16:33:41 on dist-test-slave-dssx"
server_key: "ac3620738c369dfc9e791c9850bc2dd4"
server_key_iv: "257cf63d7ef6f10ffd2d1d1f54383818"
server_key_version: "encryptionkey@0"
I20260929 16:33:41.852543 23921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:41.871059 23921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:41.871955 23921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:41.872287 23921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:41.873297 23921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:41.874831 23921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:41.874966 23921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:41.875089 23921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:41.875203 23921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:41.909175 23921 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:33333
I20260929 16:33:41.909217 24047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:33333 every 8 connection(s)
I20260929 16:33:41.910651 23921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:41.916198 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 23921
I20260929 16:33:41.916388 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:33:41.916865 6050 external_mini_cluster.cc:1428] Setting key 861c0a59a61cb7d6b45336b27a9607fe
I20260929 16:33:41.920171 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46075
--builtin_ntp_servers=127.5.232.148:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:33:41.924850 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:41.913463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:60387 (local address 127.5.232.190:46075)
0929 16:33:41.913672 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:41.913679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:41.915106 (+ 1427us) server_negotiation.cc:407] Connection header received
0929 16:33:41.917153 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:41.917159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:41.917265 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:41.917422 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:41.918845 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:41.919883 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:41.921599 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:41.922354 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:41.922495 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:41.923917 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:41.923926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:41.923934 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:41.923964 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:41.924011 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:41.924031 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:41.924122 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:41.924338 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 16:33:41.924455 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:33:41.925813 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:41.914047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46075 (local address 127.5.232.130:60387)
0929 16:33:41.914942 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:41.914978 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:41.916825 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:41.917723 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:41.917735 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:41.917850 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:41.918655 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:41.918681 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:41.920033 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:41.920037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:41.921449 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:41.921462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:41.922565 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:41.922569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:41.922768 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:41.923553 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:33:41.923556 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:41.923599 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:33:41.923679 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:41.924124 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:41.924132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:41.924138 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:33:41.924290 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 16:33:41.924604 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":263,"threads_started":1}
I20260929 16:33:41.927384 24048 heartbeater.cc:347] Connected to a master server at 127.5.232.190:46075
I20260929 16:33:41.927699 24048 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:41.928495 24048 heartbeater.cc:511] Master 127.5.232.190:46075 requested a full tablet report, sending...
I20260929 16:33:41.930028 23731 ts_manager.cc:194] Registered new tserver with Master: 4c85203c2fe84d6da619d7584274f9e9 (127.5.232.130:33333)
I20260929 16:33:41.930855 23731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.130:60387
W20260929 16:33:42.262053 24051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:42.262344 24051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:42.262518 24051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:42.269996 24051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:42.270107 24051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:42.270308 24051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:33:42.279597 24051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:46075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:42.280928 24051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:42.282778 24051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:42.296396 24057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:42.296432 24059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:42.296452 24056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:42.297605 24051 server_base.cc:1034] running on GCE node
I20260929 16:33:42.298285 24051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:42.299284 24051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:42.300489 24051 hybrid_clock.cc:648] HybridClock initialized: now 1790699622300423 us; error 68 us; skew 500 ppm
I20260929 16:33:42.300822 24051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:42.303248 24051 webserver.cc:460] Webserver started at http://127.5.232.131:36975/ using document root <none> and password file <none>
I20260929 16:33:42.303989 24051 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:42.304111 24051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:42.304450 24051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:42.307435 24051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "eae01b51c8d4418081cce316183fb6ed"
format_stamp: "Formatted at 2026-09-29 16:33:42 on dist-test-slave-dssx"
server_key: "aa6362401b3eb2fd9ee0b93c98c908c7"
server_key_iv: "38a31028e3ca53d33aeeda15b17d8149"
server_key_version: "encryptionkey@0"
I20260929 16:33:42.308267 24051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "eae01b51c8d4418081cce316183fb6ed"
format_stamp: "Formatted at 2026-09-29 16:33:42 on dist-test-slave-dssx"
server_key: "aa6362401b3eb2fd9ee0b93c98c908c7"
server_key_iv: "38a31028e3ca53d33aeeda15b17d8149"
server_key_version: "encryptionkey@0"
I20260929 16:33:42.313782 24051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 16:33:42.317507 24065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:42.318990 24051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:33:42.319200 24051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "eae01b51c8d4418081cce316183fb6ed"
format_stamp: "Formatted at 2026-09-29 16:33:42 on dist-test-slave-dssx"
server_key: "aa6362401b3eb2fd9ee0b93c98c908c7"
server_key_iv: "38a31028e3ca53d33aeeda15b17d8149"
server_key_version: "encryptionkey@0"
I20260929 16:33:42.319437 24051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:42.351956 24051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:42.352905 24051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:42.353224 24051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:42.354313 24051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:42.355860 24051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:42.355999 24051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:42.356125 24051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:42.356218 24051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:42.390498 24051 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:40229
I20260929 16:33:42.390591 24177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:40229 every 8 connection(s)
I20260929 16:33:42.391862 24051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:42.397754 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 24051
I20260929 16:33:42.397938 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:33:42.398391 6050 external_mini_cluster.cc:1428] Setting key 8049486a311498d7b4ca9316b2e322ed
I20260929 16:33:42.405335 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.394794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:43751 (local address 127.5.232.190:46075)
0929 16:33:42.395007 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.395017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.396702 (+ 1685us) server_negotiation.cc:407] Connection header received
0929 16:33:42.398771 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.398778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.398876 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.399008 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:42.400147 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.400934 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.402548 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.402958 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.403096 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.404490 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:42.404495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:42.404498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:42.404516 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:42.404549 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:42.404561 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:42.404658 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.404888 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.404990 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:33:42.406380 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.395477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46075 (local address 127.5.232.131:43751)
0929 16:33:42.396538 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.396584 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:42.398428 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:42.399273 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:42.399282 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:42.399378 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:42.399965 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.399982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.401102 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.401112 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.402393 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.402405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.403186 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.403194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.403423 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.404154 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:33:42.404156 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:42.404199 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:33:42.404277 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:42.404656 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:42.404677 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:42.404688 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:33:42.404877 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 16:33:42.405279 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":382,"threads_started":1}
I20260929 16:33:42.407693 24178 heartbeater.cc:347] Connected to a master server at 127.5.232.190:46075
I20260929 16:33:42.408003 24178 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:42.408790 24178 heartbeater.cc:511] Master 127.5.232.190:46075 requested a full tablet report, sending...
I20260929 16:33:42.410362 23731 ts_manager.cc:194] Registered new tserver with Master: eae01b51c8d4418081cce316183fb6ed (127.5.232.131:40229)
I20260929 16:33:42.411170 23731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.131:43751
I20260929 16:33:42.413650 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:42.423840 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.417935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.5.232.190:46075)
0929 16:33:42.418197 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.418206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.418595 (+ 389us) server_negotiation.cc:407] Connection header received
0929 16:33:42.418800 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.418804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.418882 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.418974 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:42.419448 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.420133 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.421554 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.421969 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.422076 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.423096 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:42.423101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:42.423104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:42.423123 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:42.423156 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:42.423169 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:42.423245 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.423412 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.423492 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:33:42.427847 23731 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:42.434934 23731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:42.468302 23917 heartbeater.cc:503] Master 127.5.232.190:46075 was elected leader, sending a full tablet report...
I20260929 16:33:42.471033 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.449731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.5.232.130:33333)
0929 16:33:42.460863 (+ 11132us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.460869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.462567 (+ 1698us) server_negotiation.cc:407] Connection header received
0929 16:33:42.462696 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.462704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.462927 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.463128 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:42.463953 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.465456 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.468027 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.469523 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.469656 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.469934 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.470494 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.470667 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11000,"thread_start_us":10906,"threads_started":1}
I20260929 16:33:42.471058 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.448651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:33333 (local address 127.0.0.1:57150)
0929 16:33:42.461870 (+ 13219us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.461890 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:42.462002 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:42.463136 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:42.463141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:42.463164 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:42.463764 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.463781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.465584 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.465591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.467829 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.467851 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.470056 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.470062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.470258 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.470296 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:33:42.470461 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 16:33:42.470667 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13090,"mutex_wait_us":91,"thread_start_us":11472,"threads_started":1}
I20260929 16:33:42.471479 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.448194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:40229 (local address 127.0.0.1:58670)
0929 16:33:42.459708 (+ 11514us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.459747 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:42.459933 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:42.462604 (+ 2671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:42.462611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:42.462642 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:42.463140 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.463153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.465194 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.465200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.468119 (+ 2919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.468135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.470458 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.470464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.470707 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.470839 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 16:33:42.470925 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:33:42.471081 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11249,"mutex_wait_us":326,"thread_start_us":11085,"threads_started":1}
I20260929 16:33:42.471616 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.448290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.5.232.129:39195)
0929 16:33:42.462529 (+ 14239us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.462540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.462567 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:33:42.462654 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.462661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.462922 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.463502 (+ 580us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:42.464008 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.465024 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.468372 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.469873 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.470011 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.470287 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.471054 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.471234 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14049,"thread_start_us":13937,"threads_started":1}
I20260929 16:33:42.472185 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.448231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39195 (local address 127.0.0.1:54142)
0929 16:33:42.460363 (+ 12132us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.460378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:42.460460 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:42.463155 (+ 2695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:42.463161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:42.463190 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:42.463818 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.463828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.465182 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.465189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.468119 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.468135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.470444 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.470451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.470710 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.470838 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 16:33:42.470925 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:33:42.471852 (+ 927us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11999,"mutex_wait_us":980,"thread_start_us":13356,"threads_started":1}
I20260929 16:33:42.472270 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.449083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.5.232.131:40229)
0929 16:33:42.461246 (+ 12163us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.461251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.461279 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:33:42.461359 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.461365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.462082 (+ 717us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.462294 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:42.463386 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.465045 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.468374 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.469872 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.470010 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.470289 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.471742 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.471903 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12025,"thread_start_us":11892,"threads_started":1}
I20260929 16:33:42.475250 23983 tablet_service.cc:1467] Processing CreateTablet for tablet d53c905dbaf24327bdcf0e26a77f4ed5 (DEFAULT_TABLE table=test-table [id=948e82f6779f4214a278c2ce0f24c77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:42.475641 23852 tablet_service.cc:1467] Processing CreateTablet for tablet d53c905dbaf24327bdcf0e26a77f4ed5 (DEFAULT_TABLE table=test-table [id=948e82f6779f4214a278c2ce0f24c77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:42.476689 23983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53c905dbaf24327bdcf0e26a77f4ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:42.476953 23852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53c905dbaf24327bdcf0e26a77f4ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:42.478817 24113 tablet_service.cc:1467] Processing CreateTablet for tablet d53c905dbaf24327bdcf0e26a77f4ed5 (DEFAULT_TABLE table=test-table [id=948e82f6779f4214a278c2ce0f24c77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:42.480338 24113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53c905dbaf24327bdcf0e26a77f4ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:42.490782 24196 tablet_bootstrap.cc:492] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6: Bootstrap starting.
I20260929 16:33:42.494438 24196 tablet_bootstrap.cc:654] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:42.495756 24196 log.cc:824] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:42.498910 24196 tablet_bootstrap.cc:492] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6: No bootstrap required, opened a new log
I20260929 16:33:42.499205 24196 ts_tablet_manager.cc:1397] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260929 16:33:42.502918 24199 tablet_bootstrap.cc:492] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: Bootstrap starting.
I20260929 16:33:42.504251 24197 tablet_bootstrap.cc:492] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: Bootstrap starting.
I20260929 16:33:42.507002 24199 tablet_bootstrap.cc:654] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:42.507071 24196 raft_consensus.cc:348] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } }
I20260929 16:33:42.507519 24196 raft_consensus.cc:374] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:42.507624 24196 raft_consensus.cc:729] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f915f27d1774200baa00b8f88e47db6, State: Initialized, Role: FOLLOWER
I20260929 16:33:42.507973 24197 tablet_bootstrap.cc:654] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:42.508279 24196 consensus_queue.cc:260] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } }
I20260929 16:33:42.508378 24199 log.cc:824] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:42.509342 24197 log.cc:824] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:42.510144 24196 ts_tablet_manager.cc:1428] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 16:33:42.513298 24199 tablet_bootstrap.cc:492] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: No bootstrap required, opened a new log
I20260929 16:33:42.513636 24199 ts_tablet_manager.cc:1397] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260929 16:33:42.513824 24197 tablet_bootstrap.cc:492] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: No bootstrap required, opened a new log
I20260929 16:33:42.514107 24197 ts_tablet_manager.cc:1397] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20260929 16:33:42.521600 24199 raft_consensus.cc:348] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } }
I20260929 16:33:42.521605 24197 raft_consensus.cc:348] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } }
I20260929 16:33:42.522029 24199 raft_consensus.cc:374] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:42.522046 24197 raft_consensus.cc:374] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:42.522142 24199 raft_consensus.cc:729] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae01b51c8d4418081cce316183fb6ed, State: Initialized, Role: FOLLOWER
I20260929 16:33:42.522161 24197 raft_consensus.cc:729] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c85203c2fe84d6da619d7584274f9e9, State: Initialized, Role: FOLLOWER
I20260929 16:33:42.522806 24199 consensus_queue.cc:260] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } }
I20260929 16:33:42.522809 24197 consensus_queue.cc:260] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } }
I20260929 16:33:42.524101 24178 heartbeater.cc:503] Master 127.5.232.190:46075 was elected leader, sending a full tablet report...
I20260929 16:33:42.524102 24048 heartbeater.cc:503] Master 127.5.232.190:46075 was elected leader, sending a full tablet report...
I20260929 16:33:42.524748 24197 ts_tablet_manager.cc:1428] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 16:33:42.524746 24199 ts_tablet_manager.cc:1428] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 16:33:42.560220 24204 raft_consensus.cc:482] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:42.560529 24204 raft_consensus.cc:504] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } }
I20260929 16:33:42.562340 24204 leader_election.cc:288] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f915f27d1774200baa00b8f88e47db6 (127.5.232.129:39195), eae01b51c8d4418081cce316183fb6ed (127.5.232.131:40229)
I20260929 16:33:42.569677 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.562856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:49979 (local address 127.5.232.129:39195)
0929 16:33:42.563085 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.563093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.563854 (+ 761us) server_negotiation.cc:407] Connection header received
0929 16:33:42.564078 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.564084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.564171 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.564290 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:42.565052 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.565862 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.567881 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.568665 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.568865 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.568946 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.569263 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.569423 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 16:33:42.569690 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.562892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:53801 (local address 127.5.232.131:40229)
0929 16:33:42.563067 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.563073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.563859 (+ 786us) server_negotiation.cc:407] Connection header received
0929 16:33:42.564078 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.564084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.564169 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.564280 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:42.564933 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.565751 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.567822 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.568648 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.568851 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.568937 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.569292 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.569450 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:33:42.569872 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.562775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:40229 (local address 127.5.232.130:53801)
0929 16:33:42.563710 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.563731 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:42.563916 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:42.564324 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:42.564329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:42.564357 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:42.564741 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.564752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.565886 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.565892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.567621 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.567635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.568868 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.568875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.569109 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.569133 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:33:42.569248 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:33:42.569404 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":108,"spinlock_wait_cycles":15232,"thread_start_us":521,"threads_started":1}
I20260929 16:33:42.569880 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.562660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39195 (local address 127.5.232.130:49979)
0929 16:33:42.563710 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.563731 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:42.563916 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:42.564457 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:42.564460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:42.564486 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:42.564893 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.564906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.565997 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.566001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.567637 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:42.567655 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.568880 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.568886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:42.569092 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.569118 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:33:42.569195 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:33:42.569360 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":47,"spinlock_wait_cycles":20608,"thread_start_us":578,"threads_started":1}
I20260929 16:33:42.570924 24133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53c905dbaf24327bdcf0e26a77f4ed5" candidate_uuid: "4c85203c2fe84d6da619d7584274f9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae01b51c8d4418081cce316183fb6ed" is_pre_election: true
I20260929 16:33:42.571599 24133 raft_consensus.cc:2454] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c85203c2fe84d6da619d7584274f9e9 in term 0.
I20260929 16:33:42.571472 23872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53c905dbaf24327bdcf0e26a77f4ed5" candidate_uuid: "4c85203c2fe84d6da619d7584274f9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f915f27d1774200baa00b8f88e47db6" is_pre_election: true
I20260929 16:33:42.572054 23872 raft_consensus.cc:2454] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c85203c2fe84d6da619d7584274f9e9 in term 0.
I20260929 16:33:42.572436 23937 leader_election.cc:302] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c85203c2fe84d6da619d7584274f9e9, eae01b51c8d4418081cce316183fb6ed; no voters:
I20260929 16:33:42.572927 24204 raft_consensus.cc:2781] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:42.573086 24204 raft_consensus.cc:482] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:42.573202 24204 raft_consensus.cc:3037] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:42.575656 24204 raft_consensus.cc:504] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } }
I20260929 16:33:42.576256 24204 leader_election.cc:288] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [CANDIDATE]: Term 1 election: Requested vote from peers 1f915f27d1774200baa00b8f88e47db6 (127.5.232.129:39195), eae01b51c8d4418081cce316183fb6ed (127.5.232.131:40229)
I20260929 16:33:42.576740 23872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53c905dbaf24327bdcf0e26a77f4ed5" candidate_uuid: "4c85203c2fe84d6da619d7584274f9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f915f27d1774200baa00b8f88e47db6"
I20260929 16:33:42.576828 24133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53c905dbaf24327bdcf0e26a77f4ed5" candidate_uuid: "4c85203c2fe84d6da619d7584274f9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae01b51c8d4418081cce316183fb6ed"
I20260929 16:33:42.576977 23872 raft_consensus.cc:3037] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:42.577036 24133 raft_consensus.cc:3037] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:42.580394 23872 raft_consensus.cc:2454] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c85203c2fe84d6da619d7584274f9e9 in term 1.
I20260929 16:33:42.580394 24133 raft_consensus.cc:2454] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c85203c2fe84d6da619d7584274f9e9 in term 1.
I20260929 16:33:42.580914 23937 leader_election.cc:302] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c85203c2fe84d6da619d7584274f9e9, eae01b51c8d4418081cce316183fb6ed; no voters:
I20260929 16:33:42.581235 24204 raft_consensus.cc:2781] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:42.581781 24204 raft_consensus.cc:686] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4c85203c2fe84d6da619d7584274f9e9, State: Running, Role: LEADER
I20260929 16:33:42.582293 24204 consensus_queue.cc:237] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } }
I20260929 16:33:42.587040 23730 catalog_manager.cc:5495] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c85203c2fe84d6da619d7584274f9e9 (127.5.232.130). New cstate: current_term: 1 leader_uuid: "4c85203c2fe84d6da619d7584274f9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c85203c2fe84d6da619d7584274f9e9" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33333 } health_report { overall_health: HEALTHY } } }
I20260929 16:33:42.616561 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.611198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.5.232.130:33333)
0929 16:33:42.611432 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.611437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.611474 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:33:42.611676 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.611681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.611778 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.611907 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:42.612242 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.612923 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.614367 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.614780 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.614881 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.615332 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:42.615940 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:42.616053 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.616129 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.616256 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:33:42.624993 23872 raft_consensus.cc:1261] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [term 1 FOLLOWER]: Refusing update from remote peer 4c85203c2fe84d6da619d7584274f9e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:42.625029 24133 raft_consensus.cc:1261] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [term 1 FOLLOWER]: Refusing update from remote peer 4c85203c2fe84d6da619d7584274f9e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:42.626123 24207 consensus_queue.cc:1035] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae01b51c8d4418081cce316183fb6ed" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:42.626468 24204 consensus_queue.cc:1035] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f915f27d1774200baa00b8f88e47db6" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:42.638402 24210 mvcc.cc:205] Tried to move back new op lower bound from 7334705654261465088 to 7334705654101041152. Current Snapshot: MvccSnapshot[applied={T|T < 7334705654261465088}]
I20260929 16:33:42.654484 23731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:42.654816 23731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:42.658833 23731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2ee070bbe374164a8d967b2405568ec: Sending DeleteTablet for 3 replicas of tablet d53c905dbaf24327bdcf0e26a77f4ed5
I20260929 16:33:42.660087 23852 tablet_service.cc:1514] Processing DeleteTablet for tablet d53c905dbaf24327bdcf0e26a77f4ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:42 UTC) from {username='slave'} at 127.0.0.1:54142
I20260929 16:33:42.660121 24113 tablet_service.cc:1514] Processing DeleteTablet for tablet d53c905dbaf24327bdcf0e26a77f4ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:42 UTC) from {username='slave'} at 127.0.0.1:58670
I20260929 16:33:42.661103 23983 tablet_service.cc:1514] Processing DeleteTablet for tablet d53c905dbaf24327bdcf0e26a77f4ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:42 UTC) from {username='slave'} at 127.0.0.1:57150
I20260929 16:33:42.665211 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 23790
I20260929 16:33:42.677330 24221 tablet_replica.cc:330] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: stopping tablet replica
I20260929 16:33:42.678223 24221 raft_consensus.cc:2229] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:33:42.679384 24221 raft_consensus.cc:2258] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:42.680565 24220 tablet_replica.cc:330] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: stopping tablet replica
I20260929 16:33:42.681092 24220 raft_consensus.cc:2229] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:42.681751 24220 raft_consensus.cc:2258] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:42.682586 24219 tablet_replica.cc:330] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6: stopping tablet replica
I20260929 16:33:42.682679 24221 ts_tablet_manager.cc:1905] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:42.683092 24219 raft_consensus.cc:2229] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:42.683629 24219 raft_consensus.cc:2258] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:42.685787 24219 ts_tablet_manager.cc:1905] T d53c905dbaf24327bdcf0e26a77f4ed5 P 1f915f27d1774200baa00b8f88e47db6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:42.686698 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.666404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.5.232.129:39195)
0929 16:33:42.666650 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.666659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.669893 (+ 3234us) server_negotiation.cc:407] Connection header received
0929 16:33:42.670871 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.670881 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.671073 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.671187 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:42.672604 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.674293 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.679576 (+ 5283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.680351 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.680475 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.684609 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:42.684622 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:42.684639 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:42.685569 (+ 930us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:42.685637 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:42.685666 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:42.685764 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.686121 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.686277 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:33:42.687062 24220 ts_tablet_manager.cc:1905] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:42.688184 23912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:42.694067 24220 ts_tablet_manager.cc:1918] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:42.694244 24220 log.cc:1196] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/d53c905dbaf24327bdcf0e26a77f4ed5
I20260929 16:33:42.694818 24220 ts_tablet_manager.cc:1939] T d53c905dbaf24327bdcf0e26a77f4ed5 P eae01b51c8d4418081cce316183fb6ed: Deleting consensus metadata
I20260929 16:33:42.696355 24221 ts_tablet_manager.cc:1918] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:42.696566 24221 log.cc:1196] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/d53c905dbaf24327bdcf0e26a77f4ed5
I20260929 16:33:42.696910 23716 catalog_manager.cc:4841] TS eae01b51c8d4418081cce316183fb6ed (127.5.232.131:40229): tablet d53c905dbaf24327bdcf0e26a77f4ed5 (table test-table [id=948e82f6779f4214a278c2ce0f24c77b]) successfully deleted
I20260929 16:33:42.697105 24221 ts_tablet_manager.cc:1939] T d53c905dbaf24327bdcf0e26a77f4ed5 P 4c85203c2fe84d6da619d7584274f9e9: Deleting consensus metadata
I20260929 16:33:42.699124 23718 catalog_manager.cc:4841] TS 4c85203c2fe84d6da619d7584274f9e9 (127.5.232.130:33333): tablet d53c905dbaf24327bdcf0e26a77f4ed5 (table test-table [id=948e82f6779f4214a278c2ce0f24c77b]) successfully deleted
I20260929 16:33:42.813987 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 23790
W20260929 16:33:42.832420 23715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.232.129:39195 (error 108)
I20260929 16:33:42.832885 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 23921
I20260929 16:33:42.834604 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.834084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39195 (local address 127.0.0.1:54158)
0929 16:33:42.834274 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.834402 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:39195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 16:33:42.834868 23715 catalog_manager.cc:4584] TS 1f915f27d1774200baa00b8f88e47db6 (127.5.232.129:39195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d53c905dbaf24327bdcf0e26a77f4ed5: Network error: Client connection negotiation failed: client connection to 127.5.232.129:39195: connect: Connection refused (error 111)
I20260929 16:33:42.841058 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.833853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.5.232.130:33333)
0929 16:33:42.834038 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.834043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.834064 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:33:42.834279 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.834287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.834403 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.834513 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:42.835038 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.835865 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.837810 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.838280 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.838380 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.839576 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:42.839585 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:42.839603 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:42.840236 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:42.840284 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:42.840304 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:42.840394 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.840560 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.840658 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:33:42.841845 24043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:42.888486 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.888001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39195 (local address 127.0.0.1:54168)
0929 16:33:42.888215 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.888319 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:39195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 16:33:42.889716 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.889336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39195 (local address 127.0.0.1:54182)
0929 16:33:42.889507 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.889592 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:39195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 16:33:42.950649 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.950080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39195 (local address 127.0.0.1:54198)
0929 16:33:42.950369 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.950466 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:39195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 16:33:42.951676 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.951363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39195 (local address 127.0.0.1:54214)
0929 16:33:42.951500 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:42.951568 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:39195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:33:42.957175 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 23921
I20260929 16:33:42.976416 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 24051
I20260929 16:33:42.982965 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:42.977242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.5.232.131:40229)
0929 16:33:42.977465 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:42.977470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:42.977490 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:33:42.977625 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:42.977629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:42.977710 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:42.977810 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:42.978146 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.978884 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.980095 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:42.980462 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:42.980557 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:42.981582 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:42.981591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:42.981605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:42.982157 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:42.982208 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:42.982227 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:42.982364 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:42.982546 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 16:33:42.982635 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:33:42.983632 24173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:43.032356 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:43.031880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39195 (local address 127.0.0.1:54230)
0929 16:33:43.032099 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:43.032189 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:39195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:33:43.033526 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:43.033070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:39195 (local address 127.0.0.1:54236)
0929 16:33:43.033272 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:43.033340 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:39195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 16:33:43.074219 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 24051
I20260929 16:33:43.091552 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 23700
I20260929 16:33:43.092710 23761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:43.205749 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 23700
2026-09-29T16:33:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2898 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:33:43 dist-test-slave-dssx krb5kdc[24230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:33:43 dist-test-slave-dssx krb5kdc[24230](info): set up 2 sockets
Sep 29 16:33:43 dist-test-slave-dssx krb5kdc[24230](info): commencing operation
krb5kdc: starting...
W20260929 16:33:44.325592 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:33:44 dist-test-slave-dssx krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:33:44.588235 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:38373
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:44.931919 24243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:44.932240 24243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:44.932427 24243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:44.939723 24243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:44.939839 24243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:44.939960 24243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:44.940076 24243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:44.940138 24243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:44.950557 24243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:44.951951 24243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:44.953356 24243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:44.965548 24248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:44.965499 24251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:44.965596 24249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:44.966482 24243 server_base.cc:1034] running on GCE node
I20260929 16:33:44.967202 24243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:44.968622 24243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:44.969820 24243 hybrid_clock.cc:648] HybridClock initialized: now 1790699624969777 us; error 32 us; skew 500 ppm
I20260929 16:33:44.970127 24243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:44 dist-test-slave-dssx krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699624, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:44.977734 24243 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:33:44.979635 24243 webserver.cc:460] Webserver started at http://127.5.232.190:39481/ using document root <none> and password file <none>
I20260929 16:33:44.980392 24243 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:44.980536 24243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:44.980904 24243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:44.984056 24243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "24cc6a4d133840499777f5ceeb06da56"
format_stamp: "Formatted at 2026-09-29 16:33:44 on dist-test-slave-dssx"
server_key: "a3ae1d1b13e60252bdfd31bf9c4f33c2"
server_key_iv: "55713e78bf7d4fa1350b77a735898a7d"
server_key_version: "encryptionkey@0"
I20260929 16:33:44.984838 24243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "24cc6a4d133840499777f5ceeb06da56"
format_stamp: "Formatted at 2026-09-29 16:33:44 on dist-test-slave-dssx"
server_key: "a3ae1d1b13e60252bdfd31bf9c4f33c2"
server_key_iv: "55713e78bf7d4fa1350b77a735898a7d"
server_key_version: "encryptionkey@0"
I20260929 16:33:44.990387 24243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:33:44.994267 24258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:44.995836 24243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:33:44.996016 24243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "24cc6a4d133840499777f5ceeb06da56"
format_stamp: "Formatted at 2026-09-29 16:33:44 on dist-test-slave-dssx"
server_key: "a3ae1d1b13e60252bdfd31bf9c4f33c2"
server_key_iv: "55713e78bf7d4fa1350b77a735898a7d"
server_key_version: "encryptionkey@0"
I20260929 16:33:44.996240 24243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:45.031342 24243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:45.035698 24243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:45.036036 24243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:45.055840 24243 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:38373
I20260929 16:33:45.055861 24309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:38373 every 8 connection(s)
I20260929 16:33:45.057359 24243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:45.061496 24310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:45.064102 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 24243
I20260929 16:33:45.064301 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:33:45.064870 6050 external_mini_cluster.cc:1428] Setting key 8984373139cc287897d71b95b66519e8
I20260929 16:33:45.075601 24310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56: Bootstrap starting.
I20260929 16:33:45.079917 24310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56: Neither blocks nor log segments found. Creating new log.
Sep 29 16:33:45 dist-test-slave-dssx krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:45.081305 24310 log.cc:824] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:45.085343 24310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56: No bootstrap required, opened a new log
I20260929 16:33:45.092052 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:45.067004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.5.232.190:38373)
0929 16:33:45.067975 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:45.067992 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:45.068042 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:33:45.069972 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:45.070001 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:45.070453 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:45.070805 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:45.072897 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:45.074424 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:45.076515 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:45.077243 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:45.077515 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:45.082929 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:45.082942 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:45.082968 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:45.083017 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:45.087056 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:45.087865 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:45.087875 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:45.087885 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:45.088121 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:45.088538 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:45.088543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:45.088547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:45.089325 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:45.089694 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:45.090080 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 16:33:45.090553 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":415,"threads_started":1}
I20260929 16:33:45.095938 24310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cc6a4d133840499777f5ceeb06da56" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38373 } }
I20260929 16:33:45.096319 24310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:45.096396 24310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24cc6a4d133840499777f5ceeb06da56, State: Initialized, Role: FOLLOWER
I20260929 16:33:45.096957 24310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cc6a4d133840499777f5ceeb06da56" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38373 } }
I20260929 16:33:45.097312 24310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:45.097553 24310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:45.097751 24310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:45.100055 24310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cc6a4d133840499777f5ceeb06da56" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38373 } }
I20260929 16:33:45.100649 24310 leader_election.cc:302] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24cc6a4d133840499777f5ceeb06da56; no voters:
I20260929 16:33:45.101312 24310 leader_election.cc:288] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:45.101476 24315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:45.103953 24310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:45.105404 24315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [term 1 LEADER]: Becoming Leader. State: Replica: 24cc6a4d133840499777f5ceeb06da56, State: Running, Role: LEADER
I20260929 16:33:45.106033 24315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cc6a4d133840499777f5ceeb06da56" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38373 } }
I20260929 16:33:45.109320 24317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24cc6a4d133840499777f5ceeb06da56. Latest consensus state: current_term: 1 leader_uuid: "24cc6a4d133840499777f5ceeb06da56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cc6a4d133840499777f5ceeb06da56" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38373 } } }
I20260929 16:33:45.109834 24317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:45.109344 24316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24cc6a4d133840499777f5ceeb06da56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cc6a4d133840499777f5ceeb06da56" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 38373 } } }
I20260929 16:33:45.110237 24316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:45.114166 24325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:45.120972 24325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:45.130879 24325 catalog_manager.cc:1347] Generated new cluster ID: 57a27668ae984be8bce8d3acaf1b4a78
I20260929 16:33:45.131048 24325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:45.148218 24325 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:45.149950 24325 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:45.164829 24325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56: Generated new TSK 0
I20260929 16:33:45.165830 24325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:33:45.231053 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:38373
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:45.578281 24336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:45.578586 24336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:45.578761 24336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:45.586455 24336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:45.586584 24336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:45.586798 24336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:33:45.597031 24336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:38373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:45.598487 24336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:45.599895 24336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:45.613912 24341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:45.613901 24342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:45.614463 24344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:45.615655 24336 server_base.cc:1034] running on GCE node
I20260929 16:33:45.616261 24336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:45.617194 24336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:45.618415 24336 hybrid_clock.cc:648] HybridClock initialized: now 1790699625618383 us; error 74 us; skew 500 ppm
I20260929 16:33:45.618755 24336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:45 dist-test-slave-dssx krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699625, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:45.622084 24336 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:33:45.623615 24336 webserver.cc:460] Webserver started at http://127.5.232.129:46217/ using document root <none> and password file <none>
I20260929 16:33:45.624367 24336 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:45.624521 24336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:45.624861 24336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:45.627899 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "84e52bd1329c478e85a16c21ff364240"
format_stamp: "Formatted at 2026-09-29 16:33:45 on dist-test-slave-dssx"
server_key: "70ed340dc0502e96c8fd1fcd80d6ad5c"
server_key_iv: "e3539556852a8804ca1150d103841a82"
server_key_version: "encryptionkey@0"
I20260929 16:33:45.628710 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "84e52bd1329c478e85a16c21ff364240"
format_stamp: "Formatted at 2026-09-29 16:33:45 on dist-test-slave-dssx"
server_key: "70ed340dc0502e96c8fd1fcd80d6ad5c"
server_key_iv: "e3539556852a8804ca1150d103841a82"
server_key_version: "encryptionkey@0"
I20260929 16:33:45.634166 24336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 16:33:45.637935 24351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:45.639348 24336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:33:45.639539 24336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "84e52bd1329c478e85a16c21ff364240"
format_stamp: "Formatted at 2026-09-29 16:33:45 on dist-test-slave-dssx"
server_key: "70ed340dc0502e96c8fd1fcd80d6ad5c"
server_key_iv: "e3539556852a8804ca1150d103841a82"
server_key_version: "encryptionkey@0"
I20260929 16:33:45.639770 24336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:45.658041 24336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:45.662437 24336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:45.662771 24336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:45.663852 24336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:45.665501 24336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:45.665608 24336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:45.665777 24336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:45.665860 24336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:45.700743 24336 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:34069
I20260929 16:33:45.700793 24463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:34069 every 8 connection(s)
I20260929 16:33:45.702198 24336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:45.706938 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 24336
I20260929 16:33:45.707142 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:33:45.707608 6050 external_mini_cluster.cc:1428] Setting key 5ac71e27ea7a04bce2d735e7aafc8776
Sep 29 16:33:45 dist-test-slave-dssx krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699625, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:45.726254 24466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:45.705078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:51839 (local address 127.5.232.190:38373)
0929 16:33:45.705624 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:45.705629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:45.706600 (+ 971us) server_negotiation.cc:407] Connection header received
0929 16:33:45.708940 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:45.708944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:45.709008 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:45.709105 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:45.710918 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:45.711645 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:45.713265 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:45.713768 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:45.713915 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:45.717708 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:45.717714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:45.717717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:45.717755 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:45.720890 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:45.722435 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:45.722443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:45.722447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:45.722638 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:45.723388 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:45.723397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:45.723403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:45.723821 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:45.723983 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:45.725618 (+ 1635us) server_negotiation.cc:299] Negotiation successful
0929 16:33:45.725876 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":304,"threads_started":1}
I20260929 16:33:45.726914 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:45.705699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:38373 (local address 127.5.232.129:51839)
0929 16:33:45.706421 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:45.706464 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:45.708607 (+ 2143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:45.709462 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:45.709479 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:45.710147 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:45.710732 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:45.710752 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:45.711827 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:45.711833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:45.713099 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:45.713111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:45.714016 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:45.714024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:45.714278 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:45.715055 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:45.715083 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:45.717461 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:45.721119 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:45.721133 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:45.721157 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:45.722235 (+ 1078us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:45.722785 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:45.722793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:45.722822 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:45.723212 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:45.724025 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:45.724034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:45.724440 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 16:33:45.724757 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 16:33:45.725376 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":269,"threads_started":1}
I20260929 16:33:45.728704 24464 heartbeater.cc:347] Connected to a master server at 127.5.232.190:38373
I20260929 16:33:45.729072 24464 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:45.730064 24464 heartbeater.cc:511] Master 127.5.232.190:38373 requested a full tablet report, sending...
I20260929 16:33:45.732547 24275 ts_manager.cc:194] Registered new tserver with Master: 84e52bd1329c478e85a16c21ff364240 (127.5.232.129:34069)
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
I20260929 16:33:45.734143 24275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:51839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:33:45.768262 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:38373
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:46.111088 24470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:46.111392 24470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:46.111486 24470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:46.118877 24470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:46.119006 24470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:46.119120 24470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:33:46.129285 24470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:38373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:46.130653 24470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:46.131982 24470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:46.145916 24478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:46.145947 24475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:46.146011 24476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:46.146874 24470 server_base.cc:1034] running on GCE node
I20260929 16:33:46.147552 24470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:46.148655 24470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:46.149897 24470 hybrid_clock.cc:648] HybridClock initialized: now 1790699626149833 us; error 58 us; skew 500 ppm
I20260929 16:33:46.150249 24470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:46 dist-test-slave-dssx krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699626, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:46.153592 24470 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:33:46.155246 24470 webserver.cc:460] Webserver started at http://127.5.232.130:45585/ using document root <none> and password file <none>
I20260929 16:33:46.155988 24470 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:46.156145 24470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:46.156489 24470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:46.159657 24470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad"
format_stamp: "Formatted at 2026-09-29 16:33:46 on dist-test-slave-dssx"
server_key: "8be3f4e35dcd5c95a6538917d9810150"
server_key_iv: "1c9d50ddeeb8dd9d574fc0fee81bebe1"
server_key_version: "encryptionkey@0"
I20260929 16:33:46.160550 24470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad"
format_stamp: "Formatted at 2026-09-29 16:33:46 on dist-test-slave-dssx"
server_key: "8be3f4e35dcd5c95a6538917d9810150"
server_key_iv: "1c9d50ddeeb8dd9d574fc0fee81bebe1"
server_key_version: "encryptionkey@0"
I20260929 16:33:46.166100 24470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:33:46.169919 24485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:46.171411 24470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:33:46.171634 24470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad"
format_stamp: "Formatted at 2026-09-29 16:33:46 on dist-test-slave-dssx"
server_key: "8be3f4e35dcd5c95a6538917d9810150"
server_key_iv: "1c9d50ddeeb8dd9d574fc0fee81bebe1"
server_key_version: "encryptionkey@0"
I20260929 16:33:46.171873 24470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:46.191577 24470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:46.195881 24470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:46.196203 24470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:46.197253 24470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:46.198755 24470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:46.198882 24470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:46.199038 24470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:46.199116 24470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:46.234493 24470 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:38047
I20260929 16:33:46.234536 24597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:38047 every 8 connection(s)
I20260929 16:33:46.235914 24470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:46.243661 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 24470
I20260929 16:33:46.243894 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:33:46.244433 6050 external_mini_cluster.cc:1428] Setting key a1c9dec977e776bf8c79a33df3ab2b7a
Sep 29 16:33:46 dist-test-slave-dssx krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699626, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:46.260273 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.238658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:49643 (local address 127.5.232.190:38373)
0929 16:33:46.239127 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:46.239131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:46.240017 (+ 886us) server_negotiation.cc:407] Connection header received
0929 16:33:46.241994 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:46.242000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:46.242101 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:46.242235 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:46.244019 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.244985 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.246702 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.247142 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.247300 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.252460 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:46.252470 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:46.252477 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:46.252541 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:46.256357 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:46.257569 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:46.257577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:46.257582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:46.257796 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:46.258322 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:46.258326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:46.258328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:46.258654 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:46.258774 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:46.259802 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0929 16:33:46.260000 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":255,"threads_started":1}
I20260929 16:33:46.261942 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.239125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:38373 (local address 127.5.232.130:49643)
0929 16:33:46.239833 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:46.239882 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:46.241695 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:46.242496 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:46.242504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:46.243137 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:46.243815 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.243844 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.245152 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.245157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.246536 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.246549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.247403 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.247410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.247659 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.248587 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:46.248630 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:46.252072 (+ 3442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:46.256578 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:46.256590 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:46.256612 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:46.257366 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:46.257954 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:46.257958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:46.257960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:46.258159 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:46.258831 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:46.258843 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:46.259356 (+ 513us) client_negotiation.cc:769] Sending connection context
0929 16:33:46.259677 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 16:33:46.260317 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":286,"threads_started":1}
I20260929 16:33:46.263597 24598 heartbeater.cc:347] Connected to a master server at 127.5.232.190:38373
I20260929 16:33:46.263935 24598 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:46.264712 24598 heartbeater.cc:511] Master 127.5.232.190:38373 requested a full tablet report, sending...
I20260929 16:33:46.266392 24275 ts_manager.cc:194] Registered new tserver with Master: e2eccb6ec84e4c7da43212a4d2b6ffad (127.5.232.130:38047)
I20260929 16:33:46.267279 24275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:49643
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:33:46.305526 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:38373
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:46.656545 24604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:46.656922 24604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:46.657109 24604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:46.665099 24604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:46.665295 24604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:46.665506 24604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:33:46.676705 24604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:38373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:46.678223 24604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:46.679646 24604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:46.694886 24610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:46.695278 24609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:46.695847 24612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:46.696733 24604 server_base.cc:1034] running on GCE node
I20260929 16:33:46.697561 24604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:46.698695 24604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:46.700081 24604 hybrid_clock.cc:648] HybridClock initialized: now 1790699626700003 us; error 68 us; skew 500 ppm
I20260929 16:33:46.700541 24604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:46 dist-test-slave-dssx krb5kdc[24230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699626, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:46.705420 24604 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:33:46.708678 24604 webserver.cc:460] Webserver started at http://127.5.232.131:46211/ using document root <none> and password file <none>
I20260929 16:33:46.709817 24604 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:46.709983 24604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:46.710382 24604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:46.715325 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "3bb52aea17fd4ad4bb24b076885aad3f"
format_stamp: "Formatted at 2026-09-29 16:33:46 on dist-test-slave-dssx"
server_key: "ce6d1ae508399fd14b756902c461fb78"
server_key_iv: "3918091e5f88f69baa3e847634ed830d"
server_key_version: "encryptionkey@0"
I20260929 16:33:46.716611 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "3bb52aea17fd4ad4bb24b076885aad3f"
format_stamp: "Formatted at 2026-09-29 16:33:46 on dist-test-slave-dssx"
server_key: "ce6d1ae508399fd14b756902c461fb78"
server_key_iv: "3918091e5f88f69baa3e847634ed830d"
server_key_version: "encryptionkey@0"
I20260929 16:33:46.725438 24604 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.005s
I20260929 16:33:46.729895 24619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:46.731630 24604 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:33:46.731858 24604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "3bb52aea17fd4ad4bb24b076885aad3f"
format_stamp: "Formatted at 2026-09-29 16:33:46 on dist-test-slave-dssx"
server_key: "ce6d1ae508399fd14b756902c461fb78"
server_key_iv: "3918091e5f88f69baa3e847634ed830d"
server_key_version: "encryptionkey@0"
I20260929 16:33:46.732091 24604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:46.737566 24464 heartbeater.cc:503] Master 127.5.232.190:38373 was elected leader, sending a full tablet report...
I20260929 16:33:46.765198 24604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:46.770009 24604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:46.770349 24604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:46.771428 24604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:46.772876 24604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:46.772992 24604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:46.773152 24604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:46.773242 24604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:46.814618 24604 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:36797
I20260929 16:33:46.814653 24731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:36797 every 8 connection(s)
I20260929 16:33:46.816064 24604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:46.816541 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 24604
I20260929 16:33:46.816763 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:33:46.817401 6050 external_mini_cluster.cc:1428] Setting key e44730cf2213b5fb615f4328ee4bd152
Sep 29 16:33:46 dist-test-slave-dssx krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699626, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:46.852146 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.819675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:41411 (local address 127.5.232.190:38373)
0929 16:33:46.831014 (+ 11339us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:46.831022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:46.831045 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:33:46.832550 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:46.832560 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:46.832680 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:46.832845 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:46.835200 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.836790 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.838662 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.839460 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.839630 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.843746 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:46.843755 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:46.843760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:46.843807 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:46.848062 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:46.848956 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:46.848964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:46.848969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:46.849158 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:46.849693 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:46.849700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:46.849706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:46.850163 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:46.850351 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:46.851285 (+ 934us) server_negotiation.cc:299] Negotiation successful
0929 16:33:46.851605 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11191,"thread_start_us":11038,"threads_started":1}
I20260929 16:33:46.852702 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.820337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:38373 (local address 127.5.232.131:41411)
0929 16:33:46.830089 (+ 9752us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:46.830145 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:46.832129 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:46.833444 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:46.833457 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:46.834199 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:46.834994 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.835014 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.836957 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.836961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.838485 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.838499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.839702 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.839707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.839913 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.840684 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:46.840714 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:46.843403 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:46.848232 (+ 4829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:46.848240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:46.848256 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:46.848804 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:46.849284 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:46.849287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:46.849290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:46.849559 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:46.850356 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:46.850362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:46.850837 (+ 475us) client_negotiation.cc:769] Sending connection context
0929 16:33:46.851086 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 16:33:46.851506 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9421,"thread_start_us":9174,"threads_started":1}
I20260929 16:33:46.854331 24732 heartbeater.cc:347] Connected to a master server at 127.5.232.190:38373
I20260929 16:33:46.854641 24732 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:46.855386 24732 heartbeater.cc:511] Master 127.5.232.190:38373 requested a full tablet report, sending...
I20260929 16:33:46.856983 24275 ts_manager.cc:194] Registered new tserver with Master: 3bb52aea17fd4ad4bb24b076885aad3f (127.5.232.131:36797)
I20260929 16:33:46.858057 24275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:41411
I20260929 16:33:46.860092 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:46.881553 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.864549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.5.232.190:38373)
0929 16:33:46.864805 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:46.864814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:46.865218 (+ 404us) server_negotiation.cc:407] Connection header received
0929 16:33:46.865403 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:46.865409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:46.865516 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:46.865653 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:46.867095 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.868451 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.869956 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.870814 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.870990 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.873898 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:46.873908 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:46.873916 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:46.873973 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:46.878084 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:46.879029 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:46.879037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:46.879043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:46.879224 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:46.879675 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:46.879683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:46.879697 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:46.880194 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:46.880521 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:46.880834 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 16:33:46.881085 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:33:46.886049 24274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:46.893115 24274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:46.948510 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.907293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34069 (local address 127.0.0.1:52794)
0929 16:33:46.939759 (+ 32466us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:46.939792 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:46.939927 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:46.940804 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:46.940809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:46.940835 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:46.941464 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.941476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.943179 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.943185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.945823 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.945835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.947365 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.947371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.947582 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.947640 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 16:33:46.947797 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 16:33:46.948010 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32349,"mutex_wait_us":94,"thread_start_us":29135,"threads_started":1}
I20260929 16:33:46.948621 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.906945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:36797 (local address 127.0.0.1:37812)
0929 16:33:46.938978 (+ 32033us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:46.939004 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:46.939150 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:46.940264 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:46.940278 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:46.940344 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:46.940799 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.940815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.942830 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.942838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.945817 (+ 2979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.945835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.947776 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.947785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.947943 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.947974 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:33:46.948063 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:33:46.948213 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31871,"mutex_wait_us":178,"thread_start_us":31727,"threads_started":1}
I20260929 16:33:46.949129 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.907980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.5.232.131:36797)
0929 16:33:46.937978 (+ 29998us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:46.937992 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:46.939130 (+ 1138us) server_negotiation.cc:407] Connection header received
0929 16:33:46.939453 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:46.939465 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:46.939772 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:46.939990 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:46.941051 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.942689 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.946022 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.947565 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.947691 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.947950 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:46.948150 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 16:33:46.948342 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29785,"thread_start_us":29617,"threads_started":1}
I20260929 16:33:46.949429 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.908476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.5.232.129:34069)
0929 16:33:46.938532 (+ 30056us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:46.938539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:46.939957 (+ 1418us) server_negotiation.cc:407] Connection header received
0929 16:33:46.940074 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:46.940082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:46.940334 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:46.940523 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:46.941769 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.943019 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.946014 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.947137 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.947331 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.947599 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:46.948860 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0929 16:33:46.949062 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29890,"thread_start_us":29724,"threads_started":1}
I20260929 16:33:46.949743 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.907461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.5.232.130:38047)
0929 16:33:46.940208 (+ 32747us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:46.940217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:46.940243 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:33:46.940364 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:46.940374 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:46.940595 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:46.940778 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:46.942680 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.944029 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.946304 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.947745 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.947869 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.949040 (+ 1171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:46.949167 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 16:33:46.949352 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32568,"thread_start_us":34210,"threads_started":1}
I20260929 16:33:46.950815 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:46.906893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:38047 (local address 127.0.0.1:57614)
0929 16:33:46.937043 (+ 30150us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:46.937113 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:46.939673 (+ 2560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:46.941055 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:46.941061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:46.941089 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:46.942484 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.942498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.944177 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.944183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.946107 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:46.946122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:46.948310 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:46.948321 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:46.948528 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:46.948566 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:33:46.948952 (+ 386us) client_negotiation.cc:240] Negotiation successful
0929 16:33:46.949102 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29737,"mutex_wait_us":64,"thread_start_us":32602,"threads_started":1}
I20260929 16:33:46.953614 24667 tablet_service.cc:1467] Processing CreateTablet for tablet e284c9233c94475fae80c2e6d370a78c (DEFAULT_TABLE table=test-table [id=a347ad22417d4588ab84d2f2863f65cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:46.953614 24399 tablet_service.cc:1467] Processing CreateTablet for tablet e284c9233c94475fae80c2e6d370a78c (DEFAULT_TABLE table=test-table [id=a347ad22417d4588ab84d2f2863f65cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:46.953972 24533 tablet_service.cc:1467] Processing CreateTablet for tablet e284c9233c94475fae80c2e6d370a78c (DEFAULT_TABLE table=test-table [id=a347ad22417d4588ab84d2f2863f65cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:46.955215 24399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e284c9233c94475fae80c2e6d370a78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:46.955233 24667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e284c9233c94475fae80c2e6d370a78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:46.955327 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e284c9233c94475fae80c2e6d370a78c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:46.968034 24751 tablet_bootstrap.cc:492] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: Bootstrap starting.
I20260929 16:33:46.971524 24751 tablet_bootstrap.cc:654] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:46.972512 24753 tablet_bootstrap.cc:492] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: Bootstrap starting.
I20260929 16:33:46.972980 24751 log.cc:824] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:46.973450 24752 tablet_bootstrap.cc:492] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: Bootstrap starting.
I20260929 16:33:46.976686 24751 tablet_bootstrap.cc:492] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: No bootstrap required, opened a new log
I20260929 16:33:46.977072 24751 ts_tablet_manager.cc:1397] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:33:46.977562 24753 tablet_bootstrap.cc:654] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:46.977555 24752 tablet_bootstrap.cc:654] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:46.978875 24752 log.cc:824] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:46.978875 24753 log.cc:824] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:46.981958 24752 tablet_bootstrap.cc:492] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: No bootstrap required, opened a new log
I20260929 16:33:46.982285 24753 tablet_bootstrap.cc:492] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: No bootstrap required, opened a new log
I20260929 16:33:46.982316 24752 ts_tablet_manager.cc:1397] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 16:33:46.982547 24753 ts_tablet_manager.cc:1397] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260929 16:33:46.987413 24751 raft_consensus.cc:348] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } }
I20260929 16:33:46.987916 24751 raft_consensus.cc:374] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:46.988008 24751 raft_consensus.cc:729] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2eccb6ec84e4c7da43212a4d2b6ffad, State: Initialized, Role: FOLLOWER
I20260929 16:33:46.988843 24751 consensus_queue.cc:260] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } }
I20260929 16:33:46.989794 24752 raft_consensus.cc:348] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } }
I20260929 16:33:46.990211 24598 heartbeater.cc:503] Master 127.5.232.190:38373 was elected leader, sending a full tablet report...
I20260929 16:33:46.990391 24752 raft_consensus.cc:374] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:46.990489 24752 raft_consensus.cc:729] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e52bd1329c478e85a16c21ff364240, State: Initialized, Role: FOLLOWER
I20260929 16:33:46.990869 24751 ts_tablet_manager.cc:1428] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260929 16:33:46.991217 24752 consensus_queue.cc:260] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } }
I20260929 16:33:46.993284 24753 raft_consensus.cc:348] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } }
I20260929 16:33:46.993820 24753 raft_consensus.cc:374] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:46.993947 24753 raft_consensus.cc:729] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb52aea17fd4ad4bb24b076885aad3f, State: Initialized, Role: FOLLOWER
I20260929 16:33:46.994997 24752 ts_tablet_manager.cc:1428] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260929 16:33:46.996394 24753 consensus_queue.cc:260] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } }
I20260929 16:33:46.997839 24732 heartbeater.cc:503] Master 127.5.232.190:38373 was elected leader, sending a full tablet report...
I20260929 16:33:46.998373 24753 ts_tablet_manager.cc:1428] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 16:33:47.065949 24759 raft_consensus.cc:482] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:47.066363 24759 raft_consensus.cc:504] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } }
I20260929 16:33:47.068528 24759 leader_election.cc:288] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2eccb6ec84e4c7da43212a4d2b6ffad (127.5.232.130:38047), 84e52bd1329c478e85a16c21ff364240 (127.5.232.129:34069)
W20260929 16:33:47.074061 24733 tablet.cc:2369] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:33:47.075510 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.068828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:38047 (local address 127.5.232.131:46057)
0929 16:33:47.069191 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.069215 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:47.069371 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:47.069941 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:47.069946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:47.069975 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:47.070184 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:47.070194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.071619 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.071624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:47.073133 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:47.073146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.074734 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.074739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:47.074929 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:47.074953 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:33:47.075075 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:33:47.075239 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260929 16:33:47.075507 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.069037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:46057 (local address 127.5.232.130:38047)
0929 16:33:47.069468 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:47.069473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:47.069492 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:33:47.069566 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:47.069572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:47.069642 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:47.069765 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:47.070352 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.071474 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.073319 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.074556 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.074738 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:47.074813 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:47.075103 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 16:33:47.075250 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260929 16:33:47.076361 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.069043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:48769 (local address 127.5.232.129:34069)
0929 16:33:47.069298 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:47.069305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:47.070297 (+ 992us) server_negotiation.cc:407] Connection header received
0929 16:33:47.070511 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:47.070515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:47.070574 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:47.070676 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:47.071184 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.072155 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.074220 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.075402 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.075592 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:47.075664 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:47.075967 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 16:33:47.076140 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:33:47.076453 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.069420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34069 (local address 127.5.232.131:48769)
0929 16:33:47.070183 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.070197 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:47.070351 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:47.070689 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:47.070693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:47.070715 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:47.071023 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:47.071032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.072326 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.072330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:47.074027 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:47.074046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.075603 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.075609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:47.075785 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:47.075815 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:33:47.075907 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:33:47.076058 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":420,"threads_started":1}
I20260929 16:33:47.076866 24553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e284c9233c94475fae80c2e6d370a78c" candidate_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" is_pre_election: true
I20260929 16:33:47.077534 24553 raft_consensus.cc:2454] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb52aea17fd4ad4bb24b076885aad3f in term 0.
I20260929 16:33:47.077627 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e284c9233c94475fae80c2e6d370a78c" candidate_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e52bd1329c478e85a16c21ff364240" is_pre_election: true
I20260929 16:33:47.078218 24419 raft_consensus.cc:2454] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb52aea17fd4ad4bb24b076885aad3f in term 0.
I20260929 16:33:47.078287 24621 leader_election.cc:302] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bb52aea17fd4ad4bb24b076885aad3f, e2eccb6ec84e4c7da43212a4d2b6ffad; no voters:
I20260929 16:33:47.078753 24759 raft_consensus.cc:2781] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:47.078933 24759 raft_consensus.cc:482] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:47.079067 24759 raft_consensus.cc:3037] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:47.082913 24759 raft_consensus.cc:504] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } }
I20260929 16:33:47.083802 24759 leader_election.cc:288] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [CANDIDATE]: Term 1 election: Requested vote from peers e2eccb6ec84e4c7da43212a4d2b6ffad (127.5.232.130:38047), 84e52bd1329c478e85a16c21ff364240 (127.5.232.129:34069)
I20260929 16:33:47.084335 24419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e284c9233c94475fae80c2e6d370a78c" candidate_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e52bd1329c478e85a16c21ff364240"
I20260929 16:33:47.084375 24553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e284c9233c94475fae80c2e6d370a78c" candidate_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad"
I20260929 16:33:47.084599 24553 raft_consensus.cc:3037] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:47.084610 24419 raft_consensus.cc:3037] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:47.086980 24419 raft_consensus.cc:2454] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb52aea17fd4ad4bb24b076885aad3f in term 1.
I20260929 16:33:47.086980 24553 raft_consensus.cc:2454] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb52aea17fd4ad4bb24b076885aad3f in term 1.
I20260929 16:33:47.087488 24622 leader_election.cc:302] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bb52aea17fd4ad4bb24b076885aad3f, 84e52bd1329c478e85a16c21ff364240; no voters:
I20260929 16:33:47.087893 24759 raft_consensus.cc:2781] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:47.088397 24759 raft_consensus.cc:686] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 1 LEADER]: Becoming Leader. State: Replica: 3bb52aea17fd4ad4bb24b076885aad3f, State: Running, Role: LEADER
I20260929 16:33:47.088923 24759 consensus_queue.cc:237] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } }
I20260929 16:33:47.094066 24275 catalog_manager.cc:5495] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bb52aea17fd4ad4bb24b076885aad3f (127.5.232.131). New cstate: current_term: 1 leader_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bb52aea17fd4ad4bb24b076885aad3f" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 } health_report { overall_health: UNKNOWN } } }
I20260929 16:33:47.116055 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.110838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.5.232.131:36797)
0929 16:33:47.111037 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:47.111042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:47.111060 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:33:47.111146 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:47.111149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:47.111224 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:47.111352 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:47.111715 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.112413 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.113871 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.114272 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.114370 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:47.114852 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:47.115350 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:47.115488 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:47.115691 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 16:33:47.115776 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:33:47.124644 24553 raft_consensus.cc:1261] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [term 1 FOLLOWER]: Refusing update from remote peer 3bb52aea17fd4ad4bb24b076885aad3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:47.124643 24419 raft_consensus.cc:1261] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [term 1 FOLLOWER]: Refusing update from remote peer 3bb52aea17fd4ad4bb24b076885aad3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:47.125716 24761 consensus_queue.cc:1035] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84e52bd1329c478e85a16c21ff364240" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:47.126112 24759 consensus_queue.cc:1035] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2eccb6ec84e4c7da43212a4d2b6ffad" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 38047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:47.138759 24766 mvcc.cc:205] Tried to move back new op lower bound from 7334705672692244480 to 7334705672560504832. Current Snapshot: MvccSnapshot[applied={T|T < 7334705672692244480}]
I20260929 16:33:47.139037 24765 mvcc.cc:205] Tried to move back new op lower bound from 7334705672692244480 to 7334705672560504832. Current Snapshot: MvccSnapshot[applied={T|T < 7334705672692244480}]
I20260929 16:33:47.152076 24275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:47.152396 24275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:47.156383 24275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24cc6a4d133840499777f5ceeb06da56: Sending DeleteTablet for 3 replicas of tablet e284c9233c94475fae80c2e6d370a78c
I20260929 16:33:47.157729 24667 tablet_service.cc:1514] Processing DeleteTablet for tablet e284c9233c94475fae80c2e6d370a78c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:47 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:37812
I20260929 16:33:47.157932 24533 tablet_service.cc:1514] Processing DeleteTablet for tablet e284c9233c94475fae80c2e6d370a78c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:47 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:57614
I20260929 16:33:47.158056 24399 tablet_service.cc:1514] Processing DeleteTablet for tablet e284c9233c94475fae80c2e6d370a78c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:47 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:52794
I20260929 16:33:47.166672 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 24336
I20260929 16:33:47.174417 24775 tablet_replica.cc:330] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: stopping tablet replica
I20260929 16:33:47.175149 24774 tablet_replica.cc:330] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: stopping tablet replica
I20260929 16:33:47.175729 24776 tablet_replica.cc:330] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: stopping tablet replica
I20260929 16:33:47.175928 24775 raft_consensus.cc:2229] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:47.176227 24776 raft_consensus.cc:2229] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:47.176581 24775 raft_consensus.cc:2258] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:47.176842 24776 raft_consensus.cc:2258] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:47.177588 24774 raft_consensus.cc:2229] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:33:47.178330 24774 raft_consensus.cc:2258] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:47.180477 24775 ts_tablet_manager.cc:1905] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:47.180483 24774 ts_tablet_manager.cc:1905] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:47.181264 24776 ts_tablet_manager.cc:1905] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:33:47 dist-test-slave-dssx krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:33:47.189330 24774 ts_tablet_manager.cc:1918] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:47.189705 24774 log.cc:1196] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/e284c9233c94475fae80c2e6d370a78c
I20260929 16:33:47.190354 24774 ts_tablet_manager.cc:1939] T e284c9233c94475fae80c2e6d370a78c P 3bb52aea17fd4ad4bb24b076885aad3f: Deleting consensus metadata
I20260929 16:33:47.192404 24259 catalog_manager.cc:4841] TS 3bb52aea17fd4ad4bb24b076885aad3f (127.5.232.131:36797): tablet e284c9233c94475fae80c2e6d370a78c (table test-table [id=a347ad22417d4588ab84d2f2863f65cf]) successfully deleted
I20260929 16:33:47.192587 24775 ts_tablet_manager.cc:1918] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:47.192742 24775 log.cc:1196] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/e284c9233c94475fae80c2e6d370a78c
I20260929 16:33:47.193562 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.168575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.5.232.129:34069)
0929 16:33:47.168817 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:47.168826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:47.168853 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:33:47.168942 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:47.168950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:47.169066 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:47.169257 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:47.175394 (+ 6137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.176689 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.178967 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.179612 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.179733 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:47.185616 (+ 5883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:47.185631 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:47.185651 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:47.185714 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:47.189712 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:47.190739 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:47.190749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:47.190753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:47.190922 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:47.191418 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:47.191424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:47.191428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:47.191903 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:47.192200 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:47.192819 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 16:33:47.193082 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:33:47.193965 24775 ts_tablet_manager.cc:1939] T e284c9233c94475fae80c2e6d370a78c P e2eccb6ec84e4c7da43212a4d2b6ffad: Deleting consensus metadata
I20260929 16:33:47.195511 24459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:47.195988 24260 catalog_manager.cc:4841] TS e2eccb6ec84e4c7da43212a4d2b6ffad (127.5.232.130:38047): tablet e284c9233c94475fae80c2e6d370a78c (table test-table [id=a347ad22417d4588ab84d2f2863f65cf]) successfully deleted
I20260929 16:33:47.288448 24776 ts_tablet_manager.cc:1918] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:47.288725 24776 log.cc:1196] T e284c9233c94475fae80c2e6d370a78c P 84e52bd1329c478e85a16c21ff364240: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/e284c9233c94475fae80c2e6d370a78c
I20260929 16:33:47.289054 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 24336
I20260929 16:33:47.313103 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 24470
W20260929 16:33:47.313143 24261 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.232.129:34069 (error 108)
I20260929 16:33:47.317119 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.316553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34069 (local address 127.0.0.1:52810)
0929 16:33:47.316790 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.316920 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 16:33:47.317467 24261 catalog_manager.cc:4584] TS 84e52bd1329c478e85a16c21ff364240 (127.5.232.129:34069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e284c9233c94475fae80c2e6d370a78c: Network error: Client connection negotiation failed: client connection to 127.5.232.129:34069: connect: Connection refused (error 111)
Sep 29 16:33:47 dist-test-slave-dssx krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:33:47.332015 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.314058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.5.232.130:38047)
0929 16:33:47.314253 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:47.314261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:47.314665 (+ 404us) server_negotiation.cc:407] Connection header received
0929 16:33:47.314749 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:47.314757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:47.314842 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:47.314970 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:47.316342 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.317250 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.318732 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.319402 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.319547 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:47.324366 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:47.324387 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:47.324420 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:47.324471 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:47.328678 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:47.329618 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:47.329627 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:47.329632 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:47.329809 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:47.330227 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:47.330232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:47.330235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:47.330773 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:47.331078 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:47.331354 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 16:33:47.331598 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:33:47.332834 24593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:47.371330 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.370717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34069 (local address 127.0.0.1:52826)
0929 16:33:47.370991 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.371117 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 16:33:47.372552 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.372123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34069 (local address 127.0.0.1:52830)
0929 16:33:47.372297 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.372407 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:33:47.427364 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 24470
I20260929 16:33:47.433334 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.432827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34069 (local address 127.0.0.1:52834)
0929 16:33:47.433055 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.433150 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 16:33:47.434480 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.434125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34069 (local address 127.0.0.1:52836)
0929 16:33:47.434287 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.434367 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 16:33:47.451145 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 24604
Sep 29 16:33:47 dist-test-slave-dssx krb5kdc[24230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:33:47.468472 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.452017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.5.232.131:36797)
0929 16:33:47.452233 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:47.452239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:47.452261 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:33:47.452403 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:47.452414 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:47.452483 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:47.452617 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:47.454052 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.454905 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.456228 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:47.456733 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:47.456895 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:47.461849 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:47.461861 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:47.461879 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:47.461924 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:47.465284 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:47.466104 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:47.466120 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:47.466123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:47.466283 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:47.466718 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:47.466724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:47.466727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:47.467133 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:47.467493 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:47.467827 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 16:33:47.468075 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:33:47.469192 24727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:47.515437 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.514831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34069 (local address 127.0.0.1:52852)
0929 16:33:47.515118 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.515234 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 16:33:47.516652 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:47.516189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34069 (local address 127.0.0.1:52854)
0929 16:33:47.516423 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:47.516513 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:34069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260929 16:33:47.593664 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 24604
I20260929 16:33:47.613813 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 24243
I20260929 16:33:47.615000 24305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:47.730214 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 24243
2026-09-29T16:33:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4524 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T16:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:47Z Disabled control of system clock
I20260929 16:33:47.812282 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:43307
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:43307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:33:48.156008 24786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:48.156337 24786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:48.156420 24786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:48.163753 24786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:48.163875 24786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:48.163913 24786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:48.163973 24786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:48.164017 24786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:48.173483 24786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:43307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:48.174798 24786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:48.176734 24786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:48.188898 24794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:48.189119 24791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:48.189100 24792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:48.189976 24786 server_base.cc:1034] running on GCE node
I20260929 16:33:48.190883 24786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:48.192483 24786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:48.193742 24786 hybrid_clock.cc:648] HybridClock initialized: now 1790699628193693 us; error 61 us; skew 500 ppm
I20260929 16:33:48.194131 24786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:48.196770 24786 webserver.cc:460] Webserver started at http://127.5.232.190:40253/ using document root <none> and password file <none>
I20260929 16:33:48.197584 24786 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:48.197731 24786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:48.198067 24786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:48.201152 24786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "7f7a4ca62d704b37ae985c834c4f48b3"
format_stamp: "Formatted at 2026-09-29 16:33:48 on dist-test-slave-dssx"
server_key: "b8941fa5113301a7c468af75b1699868"
server_key_iv: "21e1d5e760c16dbf5cd531c391d3a8ae"
server_key_version: "encryptionkey@0"
I20260929 16:33:48.202118 24786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "7f7a4ca62d704b37ae985c834c4f48b3"
format_stamp: "Formatted at 2026-09-29 16:33:48 on dist-test-slave-dssx"
server_key: "b8941fa5113301a7c468af75b1699868"
server_key_iv: "21e1d5e760c16dbf5cd531c391d3a8ae"
server_key_version: "encryptionkey@0"
I20260929 16:33:48.207978 24786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 16:33:48.211982 24800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:48.213737 24786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:48.213966 24786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "7f7a4ca62d704b37ae985c834c4f48b3"
format_stamp: "Formatted at 2026-09-29 16:33:48 on dist-test-slave-dssx"
server_key: "b8941fa5113301a7c468af75b1699868"
server_key_iv: "21e1d5e760c16dbf5cd531c391d3a8ae"
server_key_version: "encryptionkey@0"
I20260929 16:33:48.214249 24786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:48.236105 24786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:48.237257 24786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:48.237604 24786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:48.257314 24786 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:43307
I20260929 16:33:48.257386 24851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:43307 every 8 connection(s)
I20260929 16:33:48.258836 24786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:48.263018 24852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:48.268059 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 24786
I20260929 16:33:48.268270 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:33:48.268750 6050 external_mini_cluster.cc:1428] Setting key 92be358f3b192b8dee42855f9b43b242
I20260929 16:33:48.275043 24852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3: Bootstrap starting.
I20260929 16:33:48.278424 24852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:48.279556 24852 log.cc:824] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:48.283357 24852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3: No bootstrap required, opened a new log
I20260929 16:33:48.283757 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:48.270792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.5.232.190:43307)
0929 16:33:48.271897 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:48.271917 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:48.271965 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:33:48.274007 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:48.274035 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:48.274562 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:48.274998 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:48.275632 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:48.276928 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:48.278406 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:48.279210 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:48.279436 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:48.280741 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:48.280752 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:48.280768 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:48.281558 (+ 790us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:48.281664 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:48.281692 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:48.281814 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:48.282097 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 16:33:48.282459 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":399,"threads_started":1}
I20260929 16:33:48.292030 24852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a4ca62d704b37ae985c834c4f48b3" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43307 } }
I20260929 16:33:48.292423 24852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:48.292501 24852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7a4ca62d704b37ae985c834c4f48b3, State: Initialized, Role: FOLLOWER
I20260929 16:33:48.293120 24852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a4ca62d704b37ae985c834c4f48b3" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43307 } }
I20260929 16:33:48.293345 24852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:48.293529 24852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:48.293670 24852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:48.295853 24852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a4ca62d704b37ae985c834c4f48b3" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43307 } }
I20260929 16:33:48.296404 24852 leader_election.cc:302] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7a4ca62d704b37ae985c834c4f48b3; no voters:
I20260929 16:33:48.297029 24852 leader_election.cc:288] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:48.297281 24857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:48.298816 24857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7a4ca62d704b37ae985c834c4f48b3, State: Running, Role: LEADER
I20260929 16:33:48.299353 24857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a4ca62d704b37ae985c834c4f48b3" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43307 } }
I20260929 16:33:48.299542 24852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:48.302639 24858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f7a4ca62d704b37ae985c834c4f48b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a4ca62d704b37ae985c834c4f48b3" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43307 } } }
I20260929 16:33:48.302609 24859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f7a4ca62d704b37ae985c834c4f48b3. Latest consensus state: current_term: 1 leader_uuid: "7f7a4ca62d704b37ae985c834c4f48b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7a4ca62d704b37ae985c834c4f48b3" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 43307 } } }
I20260929 16:33:48.303138 24858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:48.303236 24859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:48.307083 24865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:48.312497 24865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:48.322052 24865 catalog_manager.cc:1347] Generated new cluster ID: facdffbce5884d2ea05ee68d0e2823ad
I20260929 16:33:48.322211 24865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:48.336228 24865 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:48.338294 24865 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:48.348155 24865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3: Generated new TSK 0
I20260929 16:33:48.349407 24865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:33:48.359037 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43307
--builtin_ntp_servers=127.5.232.148:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:33:48.720829 24876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:48.721189 24876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:48.721354 24876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:48.729074 24876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:48.729295 24876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:48.729513 24876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:33:48.739364 24876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:48.740823 24876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:48.742806 24876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:48.756762 24882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:48.756703 24884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:48.756798 24881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:48.757433 24876 server_base.cc:1034] running on GCE node
I20260929 16:33:48.758112 24876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:48.759176 24876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:48.760398 24876 hybrid_clock.cc:648] HybridClock initialized: now 1790699628760356 us; error 32 us; skew 500 ppm
I20260929 16:33:48.760756 24876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:48.763206 24876 webserver.cc:460] Webserver started at http://127.5.232.129:41845/ using document root <none> and password file <none>
I20260929 16:33:48.763968 24876 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:48.764096 24876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:48.764456 24876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:48.767588 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "c256d46b1d9944ffb2c99b0db23ac0b1"
format_stamp: "Formatted at 2026-09-29 16:33:48 on dist-test-slave-dssx"
server_key: "18512727fa96090fced5da4bef38e5d9"
server_key_iv: "3da740574d8f9adc84e77b944b2aecb2"
server_key_version: "encryptionkey@0"
I20260929 16:33:48.768493 24876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "c256d46b1d9944ffb2c99b0db23ac0b1"
format_stamp: "Formatted at 2026-09-29 16:33:48 on dist-test-slave-dssx"
server_key: "18512727fa96090fced5da4bef38e5d9"
server_key_iv: "3da740574d8f9adc84e77b944b2aecb2"
server_key_version: "encryptionkey@0"
I20260929 16:33:48.774151 24876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 16:33:48.778066 24890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:48.779541 24876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:33:48.779747 24876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "c256d46b1d9944ffb2c99b0db23ac0b1"
format_stamp: "Formatted at 2026-09-29 16:33:48 on dist-test-slave-dssx"
server_key: "18512727fa96090fced5da4bef38e5d9"
server_key_iv: "3da740574d8f9adc84e77b944b2aecb2"
server_key_version: "encryptionkey@0"
I20260929 16:33:48.779996 24876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:48.798197 24876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:48.799079 24876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:48.799423 24876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:48.800490 24876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:48.802165 24876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:48.802259 24876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:48.802418 24876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:48.802515 24876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:48.837249 24876 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:45621
I20260929 16:33:48.837402 25002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:45621 every 8 connection(s)
I20260929 16:33:48.838701 24876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:48.838974 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 24876
I20260929 16:33:48.839145 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:33:48.839663 6050 external_mini_cluster.cc:1428] Setting key 327b0d0dd0bc2325e4fff061c512cff3
I20260929 16:33:48.842729 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43307
--builtin_ntp_servers=127.5.232.148:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:33:48.857724 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:48.841666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:56167 (local address 127.5.232.190:43307)
0929 16:33:48.845837 (+ 4171us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:48.845845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:48.846804 (+ 959us) server_negotiation.cc:407] Connection header received
0929 16:33:48.848975 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:48.848980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:48.849072 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:48.849259 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:48.850803 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:48.851721 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:48.854469 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:48.854888 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:48.854996 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:48.856812 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:48.856817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:48.856821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:48.856843 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:48.856884 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:48.856896 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:48.856965 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:48.857264 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 16:33:48.857354 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4011,"thread_start_us":3897,"threads_started":1}
I20260929 16:33:48.859401 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:48.842602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:43307 (local address 127.5.232.129:56167)
0929 16:33:48.846589 (+ 3987us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:48.846638 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:48.848627 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:48.849550 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:48.849559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:48.849665 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:48.850595 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:48.850622 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:48.851927 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:48.851936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:48.854283 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:48.854305 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:48.855153 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:48.855162 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:48.855512 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:48.856387 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:33:48.856393 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:48.856454 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:33:48.856567 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:48.857001 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:48.857015 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:48.857026 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:33:48.857231 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 16:33:48.857762 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3355,"thread_start_us":3095,"threads_started":1}
I20260929 16:33:48.861290 25003 heartbeater.cc:347] Connected to a master server at 127.5.232.190:43307
I20260929 16:33:48.861653 25003 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:48.862479 25003 heartbeater.cc:511] Master 127.5.232.190:43307 requested a full tablet report, sending...
I20260929 16:33:48.864725 24817 ts_manager.cc:194] Registered new tserver with Master: c256d46b1d9944ffb2c99b0db23ac0b1 (127.5.232.129:45621)
I20260929 16:33:48.866314 24817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.129:56167
W20260929 16:33:49.185714 25007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:49.186051 25007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:49.186213 25007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:49.193553 25007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:49.193742 25007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:49.193917 25007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:33:49.203356 25007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:49.204736 25007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:49.206728 25007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:49.220678 25013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:49.220707 25012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:49.220795 25015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:49.221819 25007 server_base.cc:1034] running on GCE node
I20260929 16:33:49.222528 25007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:49.223551 25007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:49.224782 25007 hybrid_clock.cc:648] HybridClock initialized: now 1790699629224734 us; error 35 us; skew 500 ppm
I20260929 16:33:49.225108 25007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:49.227670 25007 webserver.cc:460] Webserver started at http://127.5.232.130:44469/ using document root <none> and password file <none>
I20260929 16:33:49.228415 25007 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:49.228511 25007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:49.228804 25007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:49.232160 25007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "d226a5d461fd43e4b7e867e2b03d2493"
format_stamp: "Formatted at 2026-09-29 16:33:49 on dist-test-slave-dssx"
server_key: "f61d085911621aed9f9172ae48c67b73"
server_key_iv: "a1fdaa5a30a3f4ea55a1ff148f2e2369"
server_key_version: "encryptionkey@0"
I20260929 16:33:49.233053 25007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "d226a5d461fd43e4b7e867e2b03d2493"
format_stamp: "Formatted at 2026-09-29 16:33:49 on dist-test-slave-dssx"
server_key: "f61d085911621aed9f9172ae48c67b73"
server_key_iv: "a1fdaa5a30a3f4ea55a1ff148f2e2369"
server_key_version: "encryptionkey@0"
I20260929 16:33:49.239087 25007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:33:49.243058 25021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:49.244567 25007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:33:49.244725 25007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "d226a5d461fd43e4b7e867e2b03d2493"
format_stamp: "Formatted at 2026-09-29 16:33:49 on dist-test-slave-dssx"
server_key: "f61d085911621aed9f9172ae48c67b73"
server_key_iv: "a1fdaa5a30a3f4ea55a1ff148f2e2369"
server_key_version: "encryptionkey@0"
I20260929 16:33:49.244940 25007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:49.261355 25007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:49.262422 25007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:49.262768 25007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:49.264047 25007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:49.266176 25007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:49.266314 25007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:49.266418 25007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:49.266495 25007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:49.303866 25007 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:43959
I20260929 16:33:49.303897 25133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:43959 every 8 connection(s)
I20260929 16:33:49.305346 25007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:49.312420 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 25007
I20260929 16:33:49.312628 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:33:49.313108 6050 external_mini_cluster.cc:1428] Setting key dc3722733b4830c7b5bb588462ec5159
I20260929 16:33:49.316468 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43307
--builtin_ntp_servers=127.5.232.148:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:33:49.319964 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.308282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:56007 (local address 127.5.232.190:43307)
0929 16:33:49.308474 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:49.308478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:49.310036 (+ 1558us) server_negotiation.cc:407] Connection header received
0929 16:33:49.311876 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:49.311883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:49.311983 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:49.312122 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:49.313455 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.314406 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.316891 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.317291 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.317453 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.319250 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:49.319255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:49.319258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:49.319276 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:49.319308 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:49.319321 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:49.319401 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:49.319669 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:33:49.319748 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:33:49.321682 25136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.308895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:43307 (local address 127.5.232.130:56007)
0929 16:33:49.309864 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:49.309911 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:49.311587 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:49.312387 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:49.312396 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:49.312520 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:49.313237 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.313258 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.314577 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.314581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.316728 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.316748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.317575 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.317584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.317971 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.318782 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:33:49.318788 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:49.318859 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:33:49.318978 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:49.319424 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:49.319439 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:49.319451 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 16:33:49.319655 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 16:33:49.320128 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":299,"threads_started":1}
I20260929 16:33:49.323320 25134 heartbeater.cc:347] Connected to a master server at 127.5.232.190:43307
I20260929 16:33:49.323647 25134 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:49.324421 25134 heartbeater.cc:511] Master 127.5.232.190:43307 requested a full tablet report, sending...
I20260929 16:33:49.326169 24817 ts_manager.cc:194] Registered new tserver with Master: d226a5d461fd43e4b7e867e2b03d2493 (127.5.232.130:43959)
I20260929 16:33:49.327193 24817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.130:56007
W20260929 16:33:49.659677 25137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:49.660037 25137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:49.660226 25137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:49.667709 25137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:49.667877 25137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:49.668071 25137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:33:49.677541 25137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:43307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:49.678949 25137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:49.680802 25137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:49.694610 25143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:49.694562 25142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:49.694645 25145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:49.695447 25137 server_base.cc:1034] running on GCE node
I20260929 16:33:49.696146 25137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:49.697252 25137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:49.698535 25137 hybrid_clock.cc:648] HybridClock initialized: now 1790699629698477 us; error 46 us; skew 500 ppm
I20260929 16:33:49.698915 25137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:49.701702 25137 webserver.cc:460] Webserver started at http://127.5.232.131:37131/ using document root <none> and password file <none>
I20260929 16:33:49.702442 25137 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:49.702564 25137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:49.702913 25137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:49.706094 25137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "584ea842891449658b00bd00a72dc635"
format_stamp: "Formatted at 2026-09-29 16:33:49 on dist-test-slave-dssx"
server_key: "f22a6615ff85c17b47141eed0bd5cd1d"
server_key_iv: "dab6a284516d62833784507bc27fa12f"
server_key_version: "encryptionkey@0"
I20260929 16:33:49.707038 25137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "584ea842891449658b00bd00a72dc635"
format_stamp: "Formatted at 2026-09-29 16:33:49 on dist-test-slave-dssx"
server_key: "f22a6615ff85c17b47141eed0bd5cd1d"
server_key_iv: "dab6a284516d62833784507bc27fa12f"
server_key_version: "encryptionkey@0"
I20260929 16:33:49.712582 25137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:33:49.716311 25151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:49.717798 25137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:49.718019 25137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "584ea842891449658b00bd00a72dc635"
format_stamp: "Formatted at 2026-09-29 16:33:49 on dist-test-slave-dssx"
server_key: "f22a6615ff85c17b47141eed0bd5cd1d"
server_key_iv: "dab6a284516d62833784507bc27fa12f"
server_key_version: "encryptionkey@0"
I20260929 16:33:49.718276 25137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:49.735051 25137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:49.735927 25137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:49.736251 25137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:49.737248 25137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:49.738819 25137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:49.738960 25137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:49.739104 25137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:49.739200 25137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:49.773227 25137 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:36447
I20260929 16:33:49.773276 25263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:36447 every 8 connection(s)
I20260929 16:33:49.774663 25137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:49.785225 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 25137
I20260929 16:33:49.785455 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:33:49.786000 6050 external_mini_cluster.cc:1428] Setting key d8004c3fd5afeb516d3e34c721ffe737
I20260929 16:33:49.789876 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.777327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:46879 (local address 127.5.232.190:43307)
0929 16:33:49.777553 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:49.777558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:49.778961 (+ 1403us) server_negotiation.cc:407] Connection header received
0929 16:33:49.780721 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:49.780725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:49.780801 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:49.780903 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:49.782567 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.783530 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.785871 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.786466 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.786594 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.788722 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:49.788732 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:49.788736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:49.788761 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:49.788833 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:49.788852 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:49.788996 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:49.789368 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 16:33:49.789511 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 16:33:49.790850 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.777895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:43307 (local address 127.5.232.131:46879)
0929 16:33:49.778805 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:49.778841 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:49.780448 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:49.781124 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:49.781132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:49.781293 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:49.782302 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.782329 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.783758 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.783762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.785651 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.785669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.786926 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.786933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.787202 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.788183 (+ 981us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:33:49.788189 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:49.788251 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:33:49.788352 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:49.789013 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:49.789025 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:49.789034 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:33:49.789324 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 16:33:49.789744 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":275,"threads_started":1}
I20260929 16:33:49.792246 25264 heartbeater.cc:347] Connected to a master server at 127.5.232.190:43307
I20260929 16:33:49.792541 25264 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:49.793282 25264 heartbeater.cc:511] Master 127.5.232.190:43307 requested a full tablet report, sending...
I20260929 16:33:49.794900 24817 ts_manager.cc:194] Registered new tserver with Master: 584ea842891449658b00bd00a72dc635 (127.5.232.131:36447)
I20260929 16:33:49.795723 24817 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.131:46879
I20260929 16:33:49.802395 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:49.812605 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.806615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.5.232.190:43307)
0929 16:33:49.806827 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:49.806831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:49.807292 (+ 461us) server_negotiation.cc:407] Connection header received
0929 16:33:49.807505 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:49.807508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:49.807582 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:49.807671 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:49.808169 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.808866 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.810368 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.810778 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.810873 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.811986 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:49.811991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:49.811994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:49.812011 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:49.812040 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:49.812051 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:49.812154 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:49.812313 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 16:33:49.812376 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:33:49.816901 24817 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:49.824426 24817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:49.864887 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.837936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.5.232.131:36447)
0929 16:33:49.853200 (+ 15264us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:49.853213 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:49.853261 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:33:49.853426 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:49.853438 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:49.853769 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:49.855217 (+ 1448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:49.855544 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.859033 (+ 3489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.862050 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.863533 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.863689 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.864013 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:49.864291 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:33:49.864483 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15035,"thread_start_us":16003,"threads_started":1}
I20260929 16:33:49.864931 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.837277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:36447 (local address 127.0.0.1:40244)
0929 16:33:49.847134 (+ 9857us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:49.847172 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:49.847361 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:49.854331 (+ 6970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:49.854344 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:49.854392 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:49.854938 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.854955 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.859164 (+ 4209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.859171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.861817 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.861833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.863741 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.863748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.864005 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.864062 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 16:33:49.864218 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 16:33:49.864527 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9607,"mutex_wait_us":67,"thread_start_us":9445,"threads_started":1}
I20260929 16:33:49.866389 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.838053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:45621 (local address 127.0.0.1:41696)
0929 16:33:49.857972 (+ 19919us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:49.858008 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:49.858149 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:49.859250 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:49.859254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:49.859295 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:49.859709 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.859720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.861705 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.861712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.863231 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.863244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.865479 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.865486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.865680 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.865705 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:33:49.865824 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:33:49.865991 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19728,"mutex_wait_us":1,"thread_start_us":20794,"threads_started":1}
I20260929 16:33:49.866425 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.838689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.5.232.129:45621)
0929 16:33:49.858557 (+ 19868us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:49.858566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:49.858602 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:33:49.858716 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:49.858728 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:49.859024 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:49.859257 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:49.859911 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.861539 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.863434 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.864846 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.864995 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.865312 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:49.865889 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 16:33:49.866059 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19489,"thread_start_us":19346,"threads_started":1}
I20260929 16:33:49.869073 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.847857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:43959 (local address 127.0.0.1:52276)
0929 16:33:49.859114 (+ 11257us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:49.859136 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:49.859258 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:49.861350 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:49.861355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:49.861405 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:49.861938 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.861954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.863817 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.863822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.866007 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:49.866023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.868160 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.868167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:49.868343 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.868484 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 16:33:49.868559 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 16:33:49.868685 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11124,"mutex_wait_us":54,"thread_start_us":9748,"threads_started":1}
I20260929 16:33:49.869464 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:49.847877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.5.232.130:43959)
0929 16:33:49.858792 (+ 10915us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:49.858806 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:49.859403 (+ 597us) server_negotiation.cc:407] Connection header received
0929 16:33:49.859524 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:49.859532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:49.859716 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:49.861162 (+ 1446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:49.862135 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.863638 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.866236 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:49.867667 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:49.867810 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:49.868023 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:49.868875 (+ 852us) server_negotiation.cc:299] Negotiation successful
0929 16:33:49.869046 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10711,"thread_start_us":10583,"threads_started":1}
I20260929 16:33:49.869617 25003 heartbeater.cc:503] Master 127.5.232.190:43307 was elected leader, sending a full tablet report...
I20260929 16:33:49.869590 25199 tablet_service.cc:1467] Processing CreateTablet for tablet 275e1f602e45441ea1d734697328a0d3 (DEFAULT_TABLE table=test-table [id=55017db91ca84153a8a90786df996995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:49.870366 24938 tablet_service.cc:1467] Processing CreateTablet for tablet 275e1f602e45441ea1d734697328a0d3 (DEFAULT_TABLE table=test-table [id=55017db91ca84153a8a90786df996995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:49.871371 25199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275e1f602e45441ea1d734697328a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:49.871821 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275e1f602e45441ea1d734697328a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:49.873625 25069 tablet_service.cc:1467] Processing CreateTablet for tablet 275e1f602e45441ea1d734697328a0d3 (DEFAULT_TABLE table=test-table [id=55017db91ca84153a8a90786df996995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:49.875007 25069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275e1f602e45441ea1d734697328a0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:49.883168 25282 tablet_bootstrap.cc:492] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: Bootstrap starting.
I20260929 16:33:49.886664 25282 tablet_bootstrap.cc:654] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:49.889889 25282 log.cc:824] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:49.894461 25285 tablet_bootstrap.cc:492] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1: Bootstrap starting.
I20260929 16:33:49.895306 25284 tablet_bootstrap.cc:492] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: Bootstrap starting.
I20260929 16:33:49.896605 25282 tablet_bootstrap.cc:492] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: No bootstrap required, opened a new log
I20260929 16:33:49.896909 25282 ts_tablet_manager.cc:1397] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260929 16:33:49.899433 25284 tablet_bootstrap.cc:654] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:49.899458 25285 tablet_bootstrap.cc:654] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:49.900866 25285 log.cc:824] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:49.901017 25284 log.cc:824] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:49.908260 25284 tablet_bootstrap.cc:492] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: No bootstrap required, opened a new log
I20260929 16:33:49.908380 25285 tablet_bootstrap.cc:492] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1: No bootstrap required, opened a new log
I20260929 16:33:49.908596 25284 ts_tablet_manager.cc:1397] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260929 16:33:49.908650 25285 ts_tablet_manager.cc:1397] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20260929 16:33:49.910159 25282 raft_consensus.cc:348] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } }
I20260929 16:33:49.910640 25282 raft_consensus.cc:374] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:49.910760 25282 raft_consensus.cc:729] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584ea842891449658b00bd00a72dc635, State: Initialized, Role: FOLLOWER
I20260929 16:33:49.911424 25282 consensus_queue.cc:260] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } }
I20260929 16:33:49.912640 25264 heartbeater.cc:503] Master 127.5.232.190:43307 was elected leader, sending a full tablet report...
I20260929 16:33:49.913347 25282 ts_tablet_manager.cc:1428] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260929 16:33:49.917685 25284 raft_consensus.cc:348] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } }
I20260929 16:33:49.918215 25284 raft_consensus.cc:374] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:49.918327 25284 raft_consensus.cc:729] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d226a5d461fd43e4b7e867e2b03d2493, State: Initialized, Role: FOLLOWER
I20260929 16:33:49.918941 25285 raft_consensus.cc:348] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } }
I20260929 16:33:49.919003 25284 consensus_queue.cc:260] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } }
I20260929 16:33:49.919337 25285 raft_consensus.cc:374] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:49.919436 25285 raft_consensus.cc:729] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c256d46b1d9944ffb2c99b0db23ac0b1, State: Initialized, Role: FOLLOWER
I20260929 16:33:49.920274 25285 consensus_queue.cc:260] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } }
I20260929 16:33:49.920708 25134 heartbeater.cc:503] Master 127.5.232.190:43307 was elected leader, sending a full tablet report...
I20260929 16:33:49.921020 25284 ts_tablet_manager.cc:1428] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 16:33:49.921995 25285 ts_tablet_manager.cc:1428] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260929 16:33:50.026598 25265 tablet.cc:2369] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:33:50.057822 25135 tablet.cc:2369] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:33:50.091337 25004 tablet.cc:2369] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:33:50.158011 25290 raft_consensus.cc:482] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:50.158401 25290 raft_consensus.cc:504] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } }
I20260929 16:33:50.160142 25290 leader_election.cc:288] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584ea842891449658b00bd00a72dc635 (127.5.232.131:36447), d226a5d461fd43e4b7e867e2b03d2493 (127.5.232.130:43959)
I20260929 16:33:50.167254 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.160693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:55425 (local address 127.5.232.131:36447)
0929 16:33:50.160883 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:50.160888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:50.161270 (+ 382us) server_negotiation.cc:407] Connection header received
0929 16:33:50.161600 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:50.161606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:50.161691 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:50.161756 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:50.162611 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.163442 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.165453 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.166267 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.166464 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:50.166559 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:50.166852 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:33:50.166988 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:33:50.167280 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.161455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:38161 (local address 127.5.232.130:43959)
0929 16:33:50.161651 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:50.161657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:50.161891 (+ 234us) server_negotiation.cc:407] Connection header received
0929 16:33:50.162060 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:50.162067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:50.162161 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:50.162261 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:50.162906 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.163621 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.165540 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.166280 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.166498 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:50.166568 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:50.166914 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 16:33:50.167041 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:33:50.167397 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.161234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:43959 (local address 127.5.232.129:38161)
0929 16:33:50.161775 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:50.161790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:50.161909 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:50.162287 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:50.162290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:50.162308 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:50.162721 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:50.162733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.163775 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.163779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:50.165287 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:50.165300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.166536 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.166542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:50.166705 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:50.166728 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:33:50.166858 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 16:33:50.166991 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":5,"thread_start_us":421,"threads_started":1}
I20260929 16:33:50.167785 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.160462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:36447 (local address 127.5.232.129:55425)
0929 16:33:50.161139 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:50.161161 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:50.161318 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:50.161976 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:50.161980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:50.162010 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:50.162449 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:50.162460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.163589 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.163596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:50.165219 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:50.165232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.166489 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.166495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:50.166681 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:50.166707 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:33:50.166784 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:33:50.166949 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":2,"thread_start_us":345,"threads_started":1}
I20260929 16:33:50.168484 25089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275e1f602e45441ea1d734697328a0d3" candidate_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d226a5d461fd43e4b7e867e2b03d2493" is_pre_election: true
I20260929 16:33:50.168736 25219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275e1f602e45441ea1d734697328a0d3" candidate_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584ea842891449658b00bd00a72dc635" is_pre_election: true
I20260929 16:33:50.169057 25089 raft_consensus.cc:2454] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c256d46b1d9944ffb2c99b0db23ac0b1 in term 0.
I20260929 16:33:50.169327 25219 raft_consensus.cc:2454] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c256d46b1d9944ffb2c99b0db23ac0b1 in term 0.
I20260929 16:33:50.169854 24891 leader_election.cc:302] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c256d46b1d9944ffb2c99b0db23ac0b1, d226a5d461fd43e4b7e867e2b03d2493; no voters:
I20260929 16:33:50.170274 25290 raft_consensus.cc:2781] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:50.170442 25290 raft_consensus.cc:482] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:50.170553 25290 raft_consensus.cc:3037] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:50.172766 25290 raft_consensus.cc:504] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } }
I20260929 16:33:50.173472 25290 leader_election.cc:288] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [CANDIDATE]: Term 1 election: Requested vote from peers 584ea842891449658b00bd00a72dc635 (127.5.232.131:36447), d226a5d461fd43e4b7e867e2b03d2493 (127.5.232.130:43959)
I20260929 16:33:50.173959 25219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275e1f602e45441ea1d734697328a0d3" candidate_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584ea842891449658b00bd00a72dc635"
I20260929 16:33:50.174031 25089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275e1f602e45441ea1d734697328a0d3" candidate_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d226a5d461fd43e4b7e867e2b03d2493"
I20260929 16:33:50.174193 25219 raft_consensus.cc:3037] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:50.174253 25089 raft_consensus.cc:3037] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:50.176435 25219 raft_consensus.cc:2454] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c256d46b1d9944ffb2c99b0db23ac0b1 in term 1.
I20260929 16:33:50.176450 25089 raft_consensus.cc:2454] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c256d46b1d9944ffb2c99b0db23ac0b1 in term 1.
I20260929 16:33:50.176896 24891 leader_election.cc:302] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584ea842891449658b00bd00a72dc635, c256d46b1d9944ffb2c99b0db23ac0b1; no voters:
I20260929 16:33:50.177246 25290 raft_consensus.cc:2781] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:50.177731 25290 raft_consensus.cc:686] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 1 LEADER]: Becoming Leader. State: Replica: c256d46b1d9944ffb2c99b0db23ac0b1, State: Running, Role: LEADER
I20260929 16:33:50.178226 25290 consensus_queue.cc:237] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } }
I20260929 16:33:50.182874 24817 catalog_manager.cc:5495] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c256d46b1d9944ffb2c99b0db23ac0b1 (127.5.232.129). New cstate: current_term: 1 leader_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c256d46b1d9944ffb2c99b0db23ac0b1" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 45621 } health_report { overall_health: HEALTHY } } }
I20260929 16:33:50.217859 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.212246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.5.232.129:45621)
0929 16:33:50.212458 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:50.212463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:50.212586 (+ 123us) server_negotiation.cc:407] Connection header received
0929 16:33:50.212766 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:50.212770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:50.212894 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:50.213028 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:50.213395 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.214106 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.215464 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.215885 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.215978 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:50.216610 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:50.217144 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:50.217353 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:50.217560 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 16:33:50.217607 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:33:50.226560 25089 raft_consensus.cc:1261] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [term 1 FOLLOWER]: Refusing update from remote peer c256d46b1d9944ffb2c99b0db23ac0b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:50.226567 25219 raft_consensus.cc:1261] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [term 1 FOLLOWER]: Refusing update from remote peer c256d46b1d9944ffb2c99b0db23ac0b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:50.227594 25293 consensus_queue.cc:1035] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "584ea842891449658b00bd00a72dc635" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:50.227977 25290 consensus_queue.cc:1035] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d226a5d461fd43e4b7e867e2b03d2493" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 43959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:50.241181 25297 mvcc.cc:205] Tried to move back new op lower bound from 7334705685397159936 to 7334705685213786112. Current Snapshot: MvccSnapshot[applied={T|T < 7334705685397159936}]
I20260929 16:33:50.248675 25298 mvcc.cc:205] Tried to move back new op lower bound from 7334705685397159936 to 7334705685213786112. Current Snapshot: MvccSnapshot[applied={T|T < 7334705685397159936}]
I20260929 16:33:50.259473 24817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:50.259757 24817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:50.263509 24817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f7a4ca62d704b37ae985c834c4f48b3: Sending DeleteTablet for 3 replicas of tablet 275e1f602e45441ea1d734697328a0d3
I20260929 16:33:50.264935 25199 tablet_service.cc:1514] Processing DeleteTablet for tablet 275e1f602e45441ea1d734697328a0d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:50 UTC) from {username='slave'} at 127.0.0.1:40244
I20260929 16:33:50.265770 25069 tablet_service.cc:1514] Processing DeleteTablet for tablet 275e1f602e45441ea1d734697328a0d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:50 UTC) from {username='slave'} at 127.0.0.1:52276
I20260929 16:33:50.266639 24938 tablet_service.cc:1514] Processing DeleteTablet for tablet 275e1f602e45441ea1d734697328a0d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:50 UTC) from {username='slave'} at 127.0.0.1:41696
I20260929 16:33:50.272071 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 24876
I20260929 16:33:50.274098 25308 tablet_replica.cc:330] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: stopping tablet replica
I20260929 16:33:50.274614 25308 raft_consensus.cc:2229] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:50.275173 25308 raft_consensus.cc:2258] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:50.275974 25309 tablet_replica.cc:330] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1: stopping tablet replica
I20260929 16:33:50.276424 25309 raft_consensus.cc:2229] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:33:50.277180 25309 raft_consensus.cc:2258] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:50.282272 25308 ts_tablet_manager.cc:1905] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:50.290822 25308 ts_tablet_manager.cc:1918] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:50.291031 25308 log.cc:1196] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/275e1f602e45441ea1d734697328a0d3
I20260929 16:33:50.291596 25308 ts_tablet_manager.cc:1939] T 275e1f602e45441ea1d734697328a0d3 P d226a5d461fd43e4b7e867e2b03d2493: Deleting consensus metadata
I20260929 16:33:50.292938 25307 tablet_replica.cc:330] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: stopping tablet replica
I20260929 16:33:50.293712 24801 catalog_manager.cc:4841] TS d226a5d461fd43e4b7e867e2b03d2493 (127.5.232.130:43959): tablet 275e1f602e45441ea1d734697328a0d3 (table test-table [id=55017db91ca84153a8a90786df996995]) successfully deleted
I20260929 16:33:50.294167 25307 raft_consensus.cc:2229] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:50.294663 25307 raft_consensus.cc:2258] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:50.294731 25309 ts_tablet_manager.cc:1905] T 275e1f602e45441ea1d734697328a0d3 P c256d46b1d9944ffb2c99b0db23ac0b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:50.297051 25307 ts_tablet_manager.cc:1905] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:50.298936 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.273776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.5.232.129:45621)
0929 16:33:50.273967 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:50.273974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:50.285631 (+ 11657us) server_negotiation.cc:407] Connection header received
0929 16:33:50.286153 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:50.286159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:50.286277 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:50.286363 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:50.287181 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.288419 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.290635 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.291302 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.291430 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:50.296612 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:50.296624 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:50.296641 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:50.297519 (+ 878us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:50.297591 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:50.297621 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:50.297739 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:50.298394 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 16:33:50.298522 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:33:50.299712 24998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:50.307145 25307 ts_tablet_manager.cc:1918] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:50.307384 25307 log.cc:1196] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/275e1f602e45441ea1d734697328a0d3
I20260929 16:33:50.308004 25307 ts_tablet_manager.cc:1939] T 275e1f602e45441ea1d734697328a0d3 P 584ea842891449658b00bd00a72dc635: Deleting consensus metadata
I20260929 16:33:50.310770 24801 catalog_manager.cc:4841] TS 584ea842891449658b00bd00a72dc635 (127.5.232.131:36447): tablet 275e1f602e45441ea1d734697328a0d3 (table test-table [id=55017db91ca84153a8a90786df996995]) successfully deleted
I20260929 16:33:50.416299 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 24876
I20260929 16:33:50.435153 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 25007
W20260929 16:33:50.435187 24802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.232.129:45621 (error 108)
I20260929 16:33:50.437254 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.436465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:45621 (local address 127.0.0.1:41718)
0929 16:33:50.436943 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:50.437063 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":259,"threads_started":1}
W20260929 16:33:50.437566 24802 catalog_manager.cc:4584] TS c256d46b1d9944ffb2c99b0db23ac0b1 (127.5.232.129:45621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 275e1f602e45441ea1d734697328a0d3: Network error: Client connection negotiation failed: client connection to 127.5.232.129:45621: connect: Connection refused (error 111)
I20260929 16:33:50.443102 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.436179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.5.232.130:43959)
0929 16:33:50.436440 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:50.436447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:50.436468 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:33:50.436569 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:50.436575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:50.436672 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:50.436824 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:50.437298 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.438259 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.439655 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.440119 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.440292 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:50.441544 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:50.441558 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:50.441582 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:50.442230 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:50.442293 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:50.442315 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:50.442476 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:50.442626 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 16:33:50.442746 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 16:33:50.443775 25129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:50.491411 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.490751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:45621 (local address 127.0.0.1:41726)
0929 16:33:50.491044 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:50.491168 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260929 16:33:50.492671 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.492207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:45621 (local address 127.0.0.1:41734)
0929 16:33:50.492408 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:50.492499 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:33:50.536921 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 25007
I20260929 16:33:50.553684 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.552965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:45621 (local address 127.0.0.1:41740)
0929 16:33:50.553272 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:50.553445 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 16:33:50.555086 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.554514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:45621 (local address 127.0.0.1:41756)
0929 16:33:50.554730 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:50.554863 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:33:50.555512 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 25137
I20260929 16:33:50.562475 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.556228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.5.232.131:36447)
0929 16:33:50.556410 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:50.556414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:50.556433 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:33:50.556514 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:50.556518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:50.556614 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:50.556699 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:50.557100 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.558087 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.559419 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:50.559813 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:50.559910 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:50.561054 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:50.561063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:50.561077 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:50.561761 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:50.561816 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:50.561839 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:50.561936 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:50.562081 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 16:33:50.562180 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:33:50.563194 25259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:50.635900 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.635409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:45621 (local address 127.0.0.1:41764)
0929 16:33:50.635643 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:50.635744 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 16:33:50.637099 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:50.636663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:45621 (local address 127.0.0.1:41768)
0929 16:33:50.636878 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:50.636959 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.129:45621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 16:33:50.654001 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 25137
I20260929 16:33:50.670878 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 24786
I20260929 16:33:50.672042 24847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:50.783989 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 24786
2026-09-29T16:33:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3052 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:33:50 dist-test-slave-dssx krb5kdc[25319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:33:50 dist-test-slave-dssx krb5kdc[25319](info): set up 2 sockets
Sep 29 16:33:50 dist-test-slave-dssx krb5kdc[25319](info): commencing operation
krb5kdc: starting...
W20260929 16:33:51.905711 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:33:52 dist-test-slave-dssx krb5kdc[25319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:33:52.164481 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:35427
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:35427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:52.504106 25332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:52.504437 25332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:52.504633 25332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:52.512081 25332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:52.512197 25332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:52.512318 25332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:52.512436 25332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:52.512517 25332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:52.522830 25332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:35427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:35427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:52.524266 25332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:52.525657 25332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:52.538090 25340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:52.538250 25338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:52.538051 25337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:52.539367 25332 server_base.cc:1034] running on GCE node
I20260929 16:33:52.540225 25332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:52.541905 25332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:52.543157 25332 hybrid_clock.cc:648] HybridClock initialized: now 1790699632543110 us; error 42 us; skew 500 ppm
I20260929 16:33:52.543520 25332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:52 dist-test-slave-dssx krb5kdc[25319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699632, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:52.546759 25332 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:33:52.548506 25332 webserver.cc:460] Webserver started at http://127.5.232.190:35383/ using document root <none> and password file <none>
I20260929 16:33:52.549270 25332 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:52.549459 25332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:52.549793 25332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:52.552906 25332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "7d19ecc86a2541dc93ce24831f1f5918"
format_stamp: "Formatted at 2026-09-29 16:33:52 on dist-test-slave-dssx"
server_key: "1189f2139ce3d32828344f115725a511"
server_key_iv: "012dafc8b23bc0ade6dfefda252341b3"
server_key_version: "encryptionkey@0"
I20260929 16:33:52.553771 25332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "7d19ecc86a2541dc93ce24831f1f5918"
format_stamp: "Formatted at 2026-09-29 16:33:52 on dist-test-slave-dssx"
server_key: "1189f2139ce3d32828344f115725a511"
server_key_iv: "012dafc8b23bc0ade6dfefda252341b3"
server_key_version: "encryptionkey@0"
I20260929 16:33:52.559366 25332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:33:52.563195 25347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:52.564673 25332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:33:52.564880 25332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "7d19ecc86a2541dc93ce24831f1f5918"
format_stamp: "Formatted at 2026-09-29 16:33:52 on dist-test-slave-dssx"
server_key: "1189f2139ce3d32828344f115725a511"
server_key_iv: "012dafc8b23bc0ade6dfefda252341b3"
server_key_version: "encryptionkey@0"
I20260929 16:33:52.565114 25332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:52.588857 25332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:52.593711 25332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:52.594012 25332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:52.612958 25332 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:35427
I20260929 16:33:52.612962 25398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:35427 every 8 connection(s)
I20260929 16:33:52.614516 25332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:52.618462 25399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:52.619966 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 25332
I20260929 16:33:52.620146 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:33:52.620600 6050 external_mini_cluster.cc:1428] Setting key 3ba3d839b6c9f902021e653b7d0f8f3b
I20260929 16:33:52.633541 25399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918: Bootstrap starting.
Sep 29 16:33:52 dist-test-slave-dssx krb5kdc[25319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:52.637045 25399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:52.638507 25399 log.cc:824] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:52.642168 25399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918: No bootstrap required, opened a new log
I20260929 16:33:52.645977 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:52.622659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.5.232.190:35427)
0929 16:33:52.623527 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:52.623543 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:52.623598 (+ 55us) server_negotiation.cc:407] Connection header received
0929 16:33:52.625467 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:52.625499 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:52.626057 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:52.626549 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:52.627928 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:52.630059 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:52.631443 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:52.632531 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:52.632719 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:52.637030 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:52.637044 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:52.637065 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:52.637121 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:52.640998 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:52.641842 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:52.641851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:52.641859 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:52.642074 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:52.642538 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:52.642543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:52.642546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:52.643289 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:52.643607 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:52.643949 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 16:33:52.644483 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":377,"threads_started":1}
I20260929 16:33:52.651996 25399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d19ecc86a2541dc93ce24831f1f5918" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35427 } }
I20260929 16:33:52.652391 25399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:52.652472 25399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d19ecc86a2541dc93ce24831f1f5918, State: Initialized, Role: FOLLOWER
I20260929 16:33:52.652994 25399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d19ecc86a2541dc93ce24831f1f5918" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35427 } }
I20260929 16:33:52.653205 25399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:52.653302 25399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:52.653518 25399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:52.656029 25399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d19ecc86a2541dc93ce24831f1f5918" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35427 } }
I20260929 16:33:52.656530 25399 leader_election.cc:302] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d19ecc86a2541dc93ce24831f1f5918; no voters:
I20260929 16:33:52.657234 25399 leader_election.cc:288] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:52.657606 25404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:52.658859 25404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [term 1 LEADER]: Becoming Leader. State: Replica: 7d19ecc86a2541dc93ce24831f1f5918, State: Running, Role: LEADER
I20260929 16:33:52.659442 25404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d19ecc86a2541dc93ce24831f1f5918" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35427 } }
I20260929 16:33:52.660023 25399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:52.662263 25406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d19ecc86a2541dc93ce24831f1f5918. Latest consensus state: current_term: 1 leader_uuid: "7d19ecc86a2541dc93ce24831f1f5918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d19ecc86a2541dc93ce24831f1f5918" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35427 } } }
I20260929 16:33:52.662891 25405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d19ecc86a2541dc93ce24831f1f5918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d19ecc86a2541dc93ce24831f1f5918" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 35427 } } }
I20260929 16:33:52.663266 25405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:52.663280 25406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918 [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:52.664338 25409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:52.670060 25409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:52.682750 25409 catalog_manager.cc:1347] Generated new cluster ID: 7daa374b1d2748ff8c27bedf7fb5fce7
I20260929 16:33:52.682896 25409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:52.700964 25409 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:52.702535 25409 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:52.709002 25409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918: Generated new TSK 0
I20260929 16:33:52.710037 25409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.232.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
Entry for principal kudu/127.5.232.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab.
I20260929 16:33:52.784293 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35427
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:53.147387 25425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:53.147691 25425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:53.147790 25425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:53.155542 25425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:53.155673 25425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:53.155869 25425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:33:53.166528 25425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.129.keytab
--principal=kudu/127.5.232.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:53.167900 25425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:53.169461 25425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:53.183760 25431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:53.183796 25433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:53.183728 25430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:53.184649 25425 server_base.cc:1034] running on GCE node
I20260929 16:33:53.185331 25425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:53.186389 25425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:53.187600 25425 hybrid_clock.cc:648] HybridClock initialized: now 1790699633187533 us; error 64 us; skew 500 ppm
I20260929 16:33:53.187896 25425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:53 dist-test-slave-dssx krb5kdc[25319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699633, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:53.191394 25425 init.cc:377] Logged in from keytab as kudu/127.5.232.129@KRBTEST.COM (short username kudu)
I20260929 16:33:53.192986 25425 webserver.cc:460] Webserver started at http://127.5.232.129:42121/ using document root <none> and password file <none>
I20260929 16:33:53.193770 25425 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:53.193912 25425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:53.194298 25425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:53.197435 25425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "8401def095b34405b817f3d192e5aade"
format_stamp: "Formatted at 2026-09-29 16:33:53 on dist-test-slave-dssx"
server_key: "c28cd6a8bb1907dcac54f29c378a5f5a"
server_key_iv: "40c8a881513a34de875ebc6703cca586"
server_key_version: "encryptionkey@0"
I20260929 16:33:53.198253 25425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "8401def095b34405b817f3d192e5aade"
format_stamp: "Formatted at 2026-09-29 16:33:53 on dist-test-slave-dssx"
server_key: "c28cd6a8bb1907dcac54f29c378a5f5a"
server_key_iv: "40c8a881513a34de875ebc6703cca586"
server_key_version: "encryptionkey@0"
I20260929 16:33:53.203768 25425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:33:53.207808 25440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:53.209493 25425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:53.209683 25425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "8401def095b34405b817f3d192e5aade"
format_stamp: "Formatted at 2026-09-29 16:33:53 on dist-test-slave-dssx"
server_key: "c28cd6a8bb1907dcac54f29c378a5f5a"
server_key_iv: "40c8a881513a34de875ebc6703cca586"
server_key_version: "encryptionkey@0"
I20260929 16:33:53.209879 25425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:53.234565 25425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:53.239451 25425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:53.239774 25425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:53.240818 25425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:53.242318 25425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:53.242391 25425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:53.242569 25425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:53.242642 25425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:53.276892 25425 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:36701
I20260929 16:33:53.276952 25552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:36701 every 8 connection(s)
I20260929 16:33:53.278498 25425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:53.286293 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 25425
I20260929 16:33:53.286515 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:33:53.287098 6050 external_mini_cluster.cc:1428] Setting key e8a6fc8291332df6867ed8b61da07570
Sep 29 16:33:53 dist-test-slave-dssx krb5kdc[25319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699633, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.129@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:53.301826 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:53.281265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:34447 (local address 127.5.232.190:35427)
0929 16:33:53.281751 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:53.281756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:53.282694 (+ 938us) server_negotiation.cc:407] Connection header received
0929 16:33:53.284627 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:53.284633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:53.284731 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:53.284866 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:53.286564 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:53.287652 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:53.289979 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:53.290700 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:53.290855 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:53.294564 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:53.294574 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:53.294582 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:53.294632 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:53.298135 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:53.299243 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:53.299248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:53.299250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:53.299384 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:53.299912 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:53.299915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:53.299918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:53.300183 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:53.300320 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:53.301278 (+ 958us) server_negotiation.cc:299] Negotiation successful
0929 16:33:53.301559 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
I20260929 16:33:53.303412 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:53.281756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35427 (local address 127.5.232.129:34447)
0929 16:33:53.282521 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:53.282564 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:53.284315 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:53.285113 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:53.285122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:53.285750 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:53.286347 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:53.286363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:53.287855 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:53.287870 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:53.289799 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:53.289813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:53.290940 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:53.290945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:53.291149 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:53.291919 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:53.291943 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:53.294274 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:53.298308 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:53.298323 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:53.298345 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:53.299087 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:53.299508 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:53.299515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:53.299522 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:53.299789 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:53.300327 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:53.300339 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:53.300849 (+ 510us) client_negotiation.cc:769] Sending connection context
0929 16:33:53.301172 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 16:33:53.301858 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":286,"threads_started":1}
I20260929 16:33:53.305138 25553 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35427
I20260929 16:33:53.305541 25553 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:53.306298 25553 heartbeater.cc:511] Master 127.5.232.190:35427 requested a full tablet report, sending...
I20260929 16:33:53.308544 25364 ts_manager.cc:194] Registered new tserver with Master: 8401def095b34405b817f3d192e5aade (127.5.232.129:36701)
I20260929 16:33:53.310134 25364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.129@KRBTEST.COM'} at 127.5.232.129:34447
WARNING: no policy specified for kudu/127.5.232.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
Entry for principal kudu/127.5.232.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab.
I20260929 16:33:53.346695 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35427
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:53.691891 25559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:53.692174 25559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:53.692363 25559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:53.700179 25559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:53.700306 25559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:53.700517 25559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:33:53.710840 25559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.130.keytab
--principal=kudu/127.5.232.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:53.712188 25559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:53.713649 25559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:53.728091 25564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:53.728017 25565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:53.729219 25567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:53.729593 25559 server_base.cc:1034] running on GCE node
I20260929 16:33:53.730296 25559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:53.731536 25559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:53.732836 25559 hybrid_clock.cc:648] HybridClock initialized: now 1790699633732763 us; error 57 us; skew 500 ppm
I20260929 16:33:53.733240 25559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:53 dist-test-slave-dssx krb5kdc[25319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699633, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:53.736934 25559 init.cc:377] Logged in from keytab as kudu/127.5.232.130@KRBTEST.COM (short username kudu)
I20260929 16:33:53.739159 25559 webserver.cc:460] Webserver started at http://127.5.232.130:43327/ using document root <none> and password file <none>
I20260929 16:33:53.739886 25559 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:53.740034 25559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:53.740355 25559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:53.743510 25559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "c547e9145fbb4556b6a7ccdc707caf5c"
format_stamp: "Formatted at 2026-09-29 16:33:53 on dist-test-slave-dssx"
server_key: "59d3363b161f617519b6e37ce59f7740"
server_key_iv: "f478bda169857199975a75f8696f2729"
server_key_version: "encryptionkey@0"
I20260929 16:33:53.744459 25559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "c547e9145fbb4556b6a7ccdc707caf5c"
format_stamp: "Formatted at 2026-09-29 16:33:53 on dist-test-slave-dssx"
server_key: "59d3363b161f617519b6e37ce59f7740"
server_key_iv: "f478bda169857199975a75f8696f2729"
server_key_version: "encryptionkey@0"
I20260929 16:33:53.750048 25559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:33:53.753751 25574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:53.755043 25559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:33:53.755219 25559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "c547e9145fbb4556b6a7ccdc707caf5c"
format_stamp: "Formatted at 2026-09-29 16:33:53 on dist-test-slave-dssx"
server_key: "59d3363b161f617519b6e37ce59f7740"
server_key_iv: "f478bda169857199975a75f8696f2729"
server_key_version: "encryptionkey@0"
I20260929 16:33:53.755429 25559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:53.776865 25559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:53.782140 25559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:53.782500 25559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:53.783860 25559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:53.785892 25559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:53.786000 25559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:53.786132 25559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:53.786199 25559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:53.835026 25559 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:37421
I20260929 16:33:53.835042 25686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:37421 every 8 connection(s)
I20260929 16:33:53.836942 25559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:53.844365 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 25559
I20260929 16:33:53.844586 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:33:53.845150 6050 external_mini_cluster.cc:1428] Setting key 73f91c113c354b5f339cc956cfb55d6a
Sep 29 16:33:53 dist-test-slave-dssx krb5kdc[25319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699633, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.130@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:53.862526 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:53.839935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:40723 (local address 127.5.232.190:35427)
0929 16:33:53.840480 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:53.840487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:53.841725 (+ 1238us) server_negotiation.cc:407] Connection header received
0929 16:33:53.845121 (+ 3396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:53.845129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:53.845228 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:53.845360 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:53.847753 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:53.848913 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:53.851093 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:53.851839 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:53.851985 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:53.855886 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:53.855896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:53.855904 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:53.855952 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:53.858865 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:53.859893 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:53.859898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:53.859900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:53.860079 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:53.860646 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:53.860650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:53.860652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:53.860947 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:53.861093 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:53.862063 (+ 970us) server_negotiation.cc:299] Negotiation successful
0929 16:33:53.862241 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":230,"threads_started":1}
I20260929 16:33:53.864061 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:53.840579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35427 (local address 127.5.232.130:40723)
0929 16:33:53.841539 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:53.841597 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:53.844753 (+ 3156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:53.845701 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:53.845717 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:53.846564 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:53.847429 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:53.847482 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:53.849083 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:53.849092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:53.850930 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:53.850944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:53.852056 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:53.852061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:53.852265 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:53.853039 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:53.853068 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:53.855605 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:53.859019 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:53.859030 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:53.859053 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:53.859730 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:53.860204 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:53.860211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:53.860218 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:53.860498 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:53.861108 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:53.861118 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:53.861655 (+ 537us) client_negotiation.cc:769] Sending connection context
0929 16:33:53.861982 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 16:33:53.862575 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":350,"threads_started":1}
I20260929 16:33:53.865712 25687 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35427
I20260929 16:33:53.866052 25687 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:53.866770 25687 heartbeater.cc:511] Master 127.5.232.190:35427 requested a full tablet report, sending...
I20260929 16:33:53.868337 25364 ts_manager.cc:194] Registered new tserver with Master: c547e9145fbb4556b6a7ccdc707caf5c (127.5.232.130:37421)
I20260929 16:33:53.869302 25364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.130@KRBTEST.COM'} at 127.5.232.130:40723
WARNING: no policy specified for kudu/127.5.232.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
Entry for principal kudu/127.5.232.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab.
I20260929 16:33:53.904035 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35427
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:54.234426 25693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:54.234697 25693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:54.234793 25693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:54.242146 25693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:54.242257 25693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:54.242388 25693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:33:54.252341 25693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.131.keytab
--principal=kudu/127.5.232.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:35427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:54.253608 25693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:54.254842 25693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:54.268590 25699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:54.268610 25701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:54.268723 25698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:54.270932 25693 server_base.cc:1034] running on GCE node
I20260929 16:33:54.271636 25693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:54.272621 25693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:54.273864 25693 hybrid_clock.cc:648] HybridClock initialized: now 1790699634273823 us; error 49 us; skew 500 ppm
I20260929 16:33:54.274199 25693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:54 dist-test-slave-dssx krb5kdc[25319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699634, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:54.277565 25693 init.cc:377] Logged in from keytab as kudu/127.5.232.131@KRBTEST.COM (short username kudu)
I20260929 16:33:54.279209 25693 webserver.cc:460] Webserver started at http://127.5.232.131:38271/ using document root <none> and password file <none>
I20260929 16:33:54.279927 25693 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:54.280058 25693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:54.280395 25693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:54.283398 25693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4"
format_stamp: "Formatted at 2026-09-29 16:33:54 on dist-test-slave-dssx"
server_key: "8d3f292ca403ca917b2091f74cd94bf0"
server_key_iv: "077cbe235fdda58685865e69a98d67fe"
server_key_version: "encryptionkey@0"
I20260929 16:33:54.284174 25693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4"
format_stamp: "Formatted at 2026-09-29 16:33:54 on dist-test-slave-dssx"
server_key: "8d3f292ca403ca917b2091f74cd94bf0"
server_key_iv: "077cbe235fdda58685865e69a98d67fe"
server_key_version: "encryptionkey@0"
I20260929 16:33:54.289285 25693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:33:54.292701 25708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:54.294173 25693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:33:54.294358 25693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4"
format_stamp: "Formatted at 2026-09-29 16:33:54 on dist-test-slave-dssx"
server_key: "8d3f292ca403ca917b2091f74cd94bf0"
server_key_iv: "077cbe235fdda58685865e69a98d67fe"
server_key_version: "encryptionkey@0"
I20260929 16:33:54.294577 25693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:54.313551 25553 heartbeater.cc:503] Master 127.5.232.190:35427 was elected leader, sending a full tablet report...
I20260929 16:33:54.330421 25693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:54.334378 25693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:54.334615 25693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:54.335551 25693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:54.336836 25693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:54.336908 25693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:54.336999 25693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:54.337044 25693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:54.370762 25693 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:36045
I20260929 16:33:54.370819 25820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:36045 every 8 connection(s)
I20260929 16:33:54.372105 25693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:54.379482 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 25693
I20260929 16:33:54.379678 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:33:54.380252 6050 external_mini_cluster.cc:1428] Setting key a71503068e29e0bb510abbdd66f361da
Sep 29 16:33:54 dist-test-slave-dssx krb5kdc[25319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699634, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.131@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:33:54.395150 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.374707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:40285 (local address 127.5.232.190:35427)
0929 16:33:54.375121 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.375126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.376237 (+ 1111us) server_negotiation.cc:407] Connection header received
0929 16:33:54.378308 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.378312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.378397 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.378539 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:54.380622 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.381662 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.383697 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.384113 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.384223 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.389125 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:54.389131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:54.389134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:54.389169 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:54.392005 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:54.392737 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:54.392741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:54.392743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:54.392874 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:54.393258 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:54.393261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:54.393263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:54.393598 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:54.393724 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:54.394593 (+ 869us) server_negotiation.cc:299] Negotiation successful
0929 16:33:54.394828 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":198,"threads_started":1}
I20260929 16:33:54.396413 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.375278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:35427 (local address 127.5.232.131:40285)
0929 16:33:54.376069 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:54.376116 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:54.378040 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:54.378824 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:54.378832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:54.379594 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:54.380432 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.380458 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.381805 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.381811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.383541 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.383554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.384388 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.384429 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.384651 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.385418 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:33:54.385464 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:54.388850 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:54.392153 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:54.392160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:54.392175 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:54.392621 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:54.392975 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:33:54.392978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:33:54.392981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:33:54.393163 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:33:54.393733 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:54.393742 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:54.394138 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 16:33:54.394461 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 16:33:54.395037 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":356,"threads_started":1}
I20260929 16:33:54.398001 25821 heartbeater.cc:347] Connected to a master server at 127.5.232.190:35427
I20260929 16:33:54.398351 25821 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:54.399086 25821 heartbeater.cc:511] Master 127.5.232.190:35427 requested a full tablet report, sending...
I20260929 16:33:54.400470 25364 ts_manager.cc:194] Registered new tserver with Master: 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 (127.5.232.131:36045)
I20260929 16:33:54.401281 25364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.232.131@KRBTEST.COM'} at 127.5.232.131:40285
I20260929 16:33:54.408339 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:54.424608 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.412240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.5.232.190:35427)
0929 16:33:54.412423 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.412429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.412839 (+ 410us) server_negotiation.cc:407] Connection header received
0929 16:33:54.412986 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.412989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.413062 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.413159 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:54.414691 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.415575 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.416829 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.417203 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.417295 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.419590 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:54.419594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:54.419598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:54.419629 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:54.422301 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:54.422990 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:54.422994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:54.422996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:54.423110 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:54.423426 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:54.423429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:54.423431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:54.423745 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:54.423952 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:54.424207 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 16:33:54.424380 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:33:54.428805 25364 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:54.435737 25364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:54.472162 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.448746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.5.232.131:36045)
0929 16:33:54.461544 (+ 12798us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.461558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.462213 (+ 655us) server_negotiation.cc:407] Connection header received
0929 16:33:54.462544 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.462552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.462831 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.463058 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:54.464845 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.466627 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.469511 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.470874 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.471035 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.471293 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:54.471611 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 16:33:54.471813 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12483,"thread_start_us":12188,"threads_started":1}
I20260929 16:33:54.472417 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.448397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.5.232.130:37421)
0929 16:33:54.462045 (+ 13648us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.462055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.462514 (+ 459us) server_negotiation.cc:407] Connection header received
0929 16:33:54.462626 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.462634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.462868 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.463059 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:54.464825 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.466274 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.469731 (+ 3457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.471000 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.471146 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.471340 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:54.471883 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 16:33:54.472065 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13462,"thread_start_us":14103,"threads_started":1}
I20260929 16:33:54.473079 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.448379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:36045 (local address 127.0.0.1:56624)
0929 16:33:54.462014 (+ 13635us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:54.462056 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:54.462307 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:54.463374 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:54.463382 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:54.463424 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:54.464079 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.464095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.466805 (+ 2710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.466810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.469200 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.469219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.471083 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.471089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.471298 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.471361 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 16:33:54.472461 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0929 16:33:54.472687 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13437,"mutex_wait_us":89,"spinlock_wait_cycles":7040,"thread_start_us":13023,"threads_started":1}
I20260929 16:33:54.473280 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.448163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:37421 (local address 127.0.0.1:52678)
0929 16:33:54.462183 (+ 14020us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:54.462201 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:54.462343 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:54.463409 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:54.463418 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:54.463457 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:54.464593 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.464611 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.466481 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.466487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.469572 (+ 3085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.469583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.471495 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.471502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.471666 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.471697 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:33:54.472672 (+ 975us) client_negotiation.cc:240] Negotiation successful
0929 16:33:54.472819 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12975,"mutex_wait_us":541,"thread_start_us":14307,"threads_started":1}
I20260929 16:33:54.474148 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.462883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:36701 (local address 127.0.0.1:40024)
0929 16:33:54.464788 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:54.464808 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:54.465324 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:54.466031 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:54.466036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:54.466060 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:54.466425 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.466439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.468162 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.468167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.470396 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.470411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.473138 (+ 2727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.473345 (+ 207us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.473563 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.473596 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:33:54.473697 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 16:33:54.473818 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1755,"mutex_wait_us":2,"thread_start_us":1569,"threads_started":1}
I20260929 16:33:54.474223 25839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.463239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.5.232.129:36701)
0929 16:33:54.464455 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.464466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.465487 (+ 1021us) server_negotiation.cc:407] Connection header received
0929 16:33:54.465594 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.465602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.465832 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.466040 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:54.466626 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.467780 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.470574 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.472873 (+ 2299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.473037 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.473268 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:54.473758 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 16:33:54.473926 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":826,"threads_started":1}
I20260929 16:33:54.477012 25622 tablet_service.cc:1467] Processing CreateTablet for tablet b40269173e6043cdb0f119f42493ebce (DEFAULT_TABLE table=test-table [id=b59863a9746a4f73ab0ec334d8e3ce66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:54.477105 25756 tablet_service.cc:1467] Processing CreateTablet for tablet b40269173e6043cdb0f119f42493ebce (DEFAULT_TABLE table=test-table [id=b59863a9746a4f73ab0ec334d8e3ce66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:54.477619 25488 tablet_service.cc:1467] Processing CreateTablet for tablet b40269173e6043cdb0f119f42493ebce (DEFAULT_TABLE table=test-table [id=b59863a9746a4f73ab0ec334d8e3ce66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:54.478556 25756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40269173e6043cdb0f119f42493ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:54.478556 25622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40269173e6043cdb0f119f42493ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:54.478928 25488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40269173e6043cdb0f119f42493ebce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:54.490986 25840 tablet_bootstrap.cc:492] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: Bootstrap starting.
I20260929 16:33:54.493635 25841 tablet_bootstrap.cc:492] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: Bootstrap starting.
I20260929 16:33:54.495200 25842 tablet_bootstrap.cc:492] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: Bootstrap starting.
I20260929 16:33:54.496400 25840 tablet_bootstrap.cc:654] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:54.497171 25841 tablet_bootstrap.cc:654] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:54.498422 25842 tablet_bootstrap.cc:654] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:54.498561 25841 log.cc:824] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:54.498562 25840 log.cc:824] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:54.499725 25842 log.cc:824] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:54.501876 25841 tablet_bootstrap.cc:492] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: No bootstrap required, opened a new log
I20260929 16:33:54.502171 25841 ts_tablet_manager.cc:1397] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:33:54.502552 25842 tablet_bootstrap.cc:492] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: No bootstrap required, opened a new log
I20260929 16:33:54.502552 25840 tablet_bootstrap.cc:492] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: No bootstrap required, opened a new log
I20260929 16:33:54.502821 25842 ts_tablet_manager.cc:1397] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 16:33:54.502810 25840 ts_tablet_manager.cc:1397] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260929 16:33:54.509902 25841 raft_consensus.cc:348] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } }
I20260929 16:33:54.510320 25841 raft_consensus.cc:374] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:54.510422 25841 raft_consensus.cc:729] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5eeef1f5b3495c8a4cd86c4b9b5dc4, State: Initialized, Role: FOLLOWER
I20260929 16:33:54.511173 25841 consensus_queue.cc:260] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } }
I20260929 16:33:54.512336 25821 heartbeater.cc:503] Master 127.5.232.190:35427 was elected leader, sending a full tablet report...
I20260929 16:33:54.513741 25841 ts_tablet_manager.cc:1428] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260929 16:33:54.513724 25840 raft_consensus.cc:348] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } }
I20260929 16:33:54.514186 25840 raft_consensus.cc:374] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:54.514310 25840 raft_consensus.cc:729] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c547e9145fbb4556b6a7ccdc707caf5c, State: Initialized, Role: FOLLOWER
I20260929 16:33:54.514243 25842 raft_consensus.cc:348] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } }
I20260929 16:33:54.514710 25842 raft_consensus.cc:374] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:54.514817 25842 raft_consensus.cc:729] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8401def095b34405b817f3d192e5aade, State: Initialized, Role: FOLLOWER
I20260929 16:33:54.514957 25840 consensus_queue.cc:260] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } }
I20260929 16:33:54.515575 25842 consensus_queue.cc:260] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } }
I20260929 16:33:54.516820 25687 heartbeater.cc:503] Master 127.5.232.190:35427 was elected leader, sending a full tablet report...
I20260929 16:33:54.516990 25840 ts_tablet_manager.cc:1428] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 16:33:54.518755 25842 ts_tablet_manager.cc:1428] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
W20260929 16:33:54.531080 25554 tablet.cc:2369] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:33:54.589336 25688 tablet.cc:2369] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:33:54.623916 25822 tablet.cc:2369] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:33:54.720342 25848 raft_consensus.cc:482] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:54.720676 25848 raft_consensus.cc:504] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } }
I20260929 16:33:54.722729 25848 leader_election.cc:288] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c547e9145fbb4556b6a7ccdc707caf5c (127.5.232.130:37421), 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 (127.5.232.131:36045)
I20260929 16:33:54.732132 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.723477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:46519 (local address 127.5.232.131:36045)
0929 16:33:54.723670 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.723677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.724028 (+ 351us) server_negotiation.cc:407] Connection header received
0929 16:33:54.724224 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.724230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.724309 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.724412 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:54.725274 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.726446 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.729664 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.730916 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.731091 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.731171 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:54.731724 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 16:33:54.731862 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:33:54.732132 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.723156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:33309 (local address 127.5.232.130:37421)
0929 16:33:54.723409 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.723419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.724027 (+ 608us) server_negotiation.cc:407] Connection header received
0929 16:33:54.724224 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.724230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.724309 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.724412 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:54.725307 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.726453 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.729671 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.730916 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.731091 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.731170 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:54.731724 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 16:33:54.731866 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:33:54.732491 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.722940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:37421 (local address 127.5.232.129:33309)
0929 16:33:54.723864 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:54.723885 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:54.724093 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:54.724459 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:54.724464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:54.724499 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:54.725172 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.725184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.726587 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.726591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.729438 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.729456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.731188 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.731202 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.731478 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.731565 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 16:33:54.731670 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:33:54.731862 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":348,"spinlock_wait_cycles":39040,"thread_start_us":730,"threads_started":1}
I20260929 16:33:54.732525 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.722971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:36045 (local address 127.5.232.129:46519)
0929 16:33:54.723905 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:54.723931 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:54.724056 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:54.724459 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:54.724464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:54.724504 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:54.725069 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.725089 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.726586 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.726591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.729438 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:54.729456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.731214 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.731219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:54.731475 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.731571 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 16:33:54.731670 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:33:54.731862 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":421,"spinlock_wait_cycles":4480,"thread_start_us":608,"threads_started":1}
I20260929 16:33:54.733640 25642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40269173e6043cdb0f119f42493ebce" candidate_uuid: "8401def095b34405b817f3d192e5aade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" is_pre_election: true
I20260929 16:33:54.733690 25776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40269173e6043cdb0f119f42493ebce" candidate_uuid: "8401def095b34405b817f3d192e5aade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" is_pre_election: true
I20260929 16:33:54.734160 25776 raft_consensus.cc:2454] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8401def095b34405b817f3d192e5aade in term 0.
I20260929 16:33:54.734160 25642 raft_consensus.cc:2454] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8401def095b34405b817f3d192e5aade in term 0.
I20260929 16:33:54.734848 25444 leader_election.cc:302] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c5eeef1f5b3495c8a4cd86c4b9b5dc4, 8401def095b34405b817f3d192e5aade, c547e9145fbb4556b6a7ccdc707caf5c; no voters:
I20260929 16:33:54.735265 25848 raft_consensus.cc:2781] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:54.735437 25848 raft_consensus.cc:482] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:54.735635 25848 raft_consensus.cc:3037] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:54.737914 25848 raft_consensus.cc:504] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } }
I20260929 16:33:54.738639 25848 leader_election.cc:288] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [CANDIDATE]: Term 1 election: Requested vote from peers c547e9145fbb4556b6a7ccdc707caf5c (127.5.232.130:37421), 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 (127.5.232.131:36045)
I20260929 16:33:54.739132 25642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40269173e6043cdb0f119f42493ebce" candidate_uuid: "8401def095b34405b817f3d192e5aade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c547e9145fbb4556b6a7ccdc707caf5c"
I20260929 16:33:54.739235 25776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40269173e6043cdb0f119f42493ebce" candidate_uuid: "8401def095b34405b817f3d192e5aade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4"
I20260929 16:33:54.739348 25642 raft_consensus.cc:3037] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:54.739480 25776 raft_consensus.cc:3037] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:54.741763 25642 raft_consensus.cc:2454] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8401def095b34405b817f3d192e5aade in term 1.
I20260929 16:33:54.741775 25776 raft_consensus.cc:2454] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8401def095b34405b817f3d192e5aade in term 1.
I20260929 16:33:54.742230 25441 leader_election.cc:302] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8401def095b34405b817f3d192e5aade, c547e9145fbb4556b6a7ccdc707caf5c; no voters:
I20260929 16:33:54.742529 25848 raft_consensus.cc:2781] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:54.742964 25848 raft_consensus.cc:686] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 1 LEADER]: Becoming Leader. State: Replica: 8401def095b34405b817f3d192e5aade, State: Running, Role: LEADER
I20260929 16:33:54.743500 25848 consensus_queue.cc:237] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } }
I20260929 16:33:54.748104 25362 catalog_manager.cc:5495] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade reported cstate change: term changed from 0 to 1, leader changed from <none> to 8401def095b34405b817f3d192e5aade (127.5.232.129). New cstate: current_term: 1 leader_uuid: "8401def095b34405b817f3d192e5aade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8401def095b34405b817f3d192e5aade" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 36701 } health_report { overall_health: HEALTHY } } }
I20260929 16:33:54.776502 25839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.771622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.5.232.129:36701)
0929 16:33:54.771812 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.771817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.771834 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:33:54.771921 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.771924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.772007 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.772134 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:54.772461 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.773156 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.774544 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.774943 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.775037 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.775505 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:54.776018 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:54.776128 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:54.776223 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 16:33:54.776291 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:33:54.785295 25642 raft_consensus.cc:1261] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [term 1 FOLLOWER]: Refusing update from remote peer 8401def095b34405b817f3d192e5aade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:54.785295 25776 raft_consensus.cc:1261] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [term 1 FOLLOWER]: Refusing update from remote peer 8401def095b34405b817f3d192e5aade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:54.786409 25851 consensus_queue.cc:1035] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c5eeef1f5b3495c8a4cd86c4b9b5dc4" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:54.786793 25848 consensus_queue.cc:1035] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [LEADER]: Connected to new peer: Peer: permanent_uuid: "c547e9145fbb4556b6a7ccdc707caf5c" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:54.795755 25854 mvcc.cc:205] Tried to move back new op lower bound from 7334705704070152192 to 7334705703913365504. Current Snapshot: MvccSnapshot[applied={T|T < 7334705704070152192}]
I20260929 16:33:54.805626 25855 mvcc.cc:205] Tried to move back new op lower bound from 7334705704070152192 to 7334705703913365504. Current Snapshot: MvccSnapshot[applied={T|T < 7334705704070152192}]
I20260929 16:33:54.820029 25362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:54.820339 25362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:54.824096 25362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d19ecc86a2541dc93ce24831f1f5918: Sending DeleteTablet for 3 replicas of tablet b40269173e6043cdb0f119f42493ebce
I20260929 16:33:54.825413 25756 tablet_service.cc:1514] Processing DeleteTablet for tablet b40269173e6043cdb0f119f42493ebce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:54 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:56624
I20260929 16:33:54.825436 25488 tablet_service.cc:1514] Processing DeleteTablet for tablet b40269173e6043cdb0f119f42493ebce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:54 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:40024
I20260929 16:33:54.826181 25622 tablet_service.cc:1514] Processing DeleteTablet for tablet b40269173e6043cdb0f119f42493ebce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:54 UTC) from {username='kudu', principal='kudu/127.5.232.190@KRBTEST.COM'} at 127.0.0.1:52678
I20260929 16:33:54.827517 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 25425
I20260929 16:33:54.837826 25866 tablet_replica.cc:330] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: stopping tablet replica
I20260929 16:33:54.838423 25866 raft_consensus.cc:2229] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:54.839064 25866 raft_consensus.cc:2258] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:54.839948 25865 tablet_replica.cc:330] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: stopping tablet replica
I20260929 16:33:54.840381 25865 raft_consensus.cc:2229] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:33:54.841281 25865 raft_consensus.cc:2258] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:54.843848 25865 ts_tablet_manager.cc:1905] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:54.845363 25867 tablet_replica.cc:330] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: stopping tablet replica
I20260929 16:33:54.845765 25867 raft_consensus.cc:2229] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:54.846239 25867 raft_consensus.cc:2258] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:54.848045 25866 ts_tablet_manager.cc:1905] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:54.848043 25867 ts_tablet_manager.cc:1905] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:33:54 dist-test-slave-dssx krb5kdc[25319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.129@KRBTEST.COM
I20260929 16:33:54.855129 25867 ts_tablet_manager.cc:1918] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:54.855311 25867 log.cc:1196] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/b40269173e6043cdb0f119f42493ebce
I20260929 16:33:54.855849 25867 ts_tablet_manager.cc:1939] T b40269173e6043cdb0f119f42493ebce P c547e9145fbb4556b6a7ccdc707caf5c: Deleting consensus metadata
I20260929 16:33:54.857800 25348 catalog_manager.cc:4841] TS c547e9145fbb4556b6a7ccdc707caf5c (127.5.232.130:37421): tablet b40269173e6043cdb0f119f42493ebce (table test-table [id=b59863a9746a4f73ab0ec334d8e3ce66]) successfully deleted
I20260929 16:33:54.858752 25865 ts_tablet_manager.cc:1918] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:54.858923 25865 log.cc:1196] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/b40269173e6043cdb0f119f42493ebce
I20260929 16:33:54.859458 25865 ts_tablet_manager.cc:1939] T b40269173e6043cdb0f119f42493ebce P 8401def095b34405b817f3d192e5aade: Deleting consensus metadata
I20260929 16:33:54.860244 25866 ts_tablet_manager.cc:1918] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:54.860409 25866 log.cc:1196] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/b40269173e6043cdb0f119f42493ebce
I20260929 16:33:54.860927 25866 ts_tablet_manager.cc:1939] T b40269173e6043cdb0f119f42493ebce P 1c5eeef1f5b3495c8a4cd86c4b9b5dc4: Deleting consensus metadata
I20260929 16:33:54.861491 25348 catalog_manager.cc:4841] TS 8401def095b34405b817f3d192e5aade (127.5.232.129:36701): tablet b40269173e6043cdb0f119f42493ebce (table test-table [id=b59863a9746a4f73ab0ec334d8e3ce66]) successfully deleted
I20260929 16:33:54.862543 25839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.829649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.5.232.129:36701)
0929 16:33:54.830144 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.830150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.840209 (+ 10059us) server_negotiation.cc:407] Connection header received
0929 16:33:54.840813 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.840819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.840901 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.841027 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:54.843017 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.844117 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.846220 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.846928 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.847047 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.853077 (+ 6030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:54.853095 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:54.853113 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:54.853155 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:54.858202 (+ 5047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:54.859380 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:54.859389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:54.859393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:54.859656 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:54.860270 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:54.860276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:54.860281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:54.860838 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:54.861182 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:54.861895 (+ 713us) server_negotiation.cc:299] Negotiation successful
0929 16:33:54.862157 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367}
I20260929 16:33:54.862588 25351 catalog_manager.cc:4841] TS 1c5eeef1f5b3495c8a4cd86c4b9b5dc4 (127.5.232.131:36045): tablet b40269173e6043cdb0f119f42493ebce (table test-table [id=b59863a9746a4f73ab0ec334d8e3ce66]) successfully deleted
I20260929 16:33:54.864181 25548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:54.966759 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 25425
I20260929 16:33:54.985275 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 25559
Sep 29 16:33:54 dist-test-slave-dssx krb5kdc[25319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.130@KRBTEST.COM
I20260929 16:33:55.001497 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:54.986123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.5.232.130:37421)
0929 16:33:54.986296 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:54.986300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:54.986319 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:33:54.986457 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:54.986461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:54.986525 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:54.986615 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:54.987889 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.988646 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.990187 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:54.990593 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:54.990702 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:54.995324 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:54.995345 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:54.995362 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:54.995406 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:54.998531 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:54.999378 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:54.999391 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:54.999397 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:54.999626 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:55.000031 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:55.000034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:55.000036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:55.000450 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:55.000691 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:55.000974 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 16:33:55.001148 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:33:55.002174 25682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:55.094398 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 25559
I20260929 16:33:55.112804 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 25693
Sep 29 16:33:55 dist-test-slave-dssx krb5kdc[25319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.131@KRBTEST.COM
I20260929 16:33:55.129212 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:55.113647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.5.232.131:36045)
0929 16:33:55.113843 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:55.113848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:55.113869 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:33:55.113986 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:55.113991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:55.114067 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:55.114156 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:55.115488 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:55.116195 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:55.117712 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:55.118113 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:55.118224 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:55.122713 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:55.122727 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:55.122744 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:55.122783 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:55.126248 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:55.126977 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:55.126985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:55.126987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:55.127124 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:55.127487 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:55.127490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:55.127492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:55.127856 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:55.128199 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:55.128610 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 16:33:55.128865 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:33:55.129944 25816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:55.219928 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 25693
I20260929 16:33:55.236456 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 25332
I20260929 16:33:55.237622 25394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:55.349677 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 25332
2026-09-29T16:33:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4565 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T16:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:55Z Disabled control of system clock
I20260929 16:33:55.428100 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42413
--webserver_interface=127.5.232.190
--webserver_port=0
--builtin_ntp_servers=127.5.232.148:41429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:42413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:33:55.774296 25876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:55.774622 25876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:55.774803 25876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:55.782262 25876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:55.782387 25876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:55.782481 25876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:55.782593 25876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:55.782662 25876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:55.792065 25876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:42413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:42413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:55.793502 25876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:55.795380 25876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:55.807605 25882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:55.807592 25881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:55.807626 25884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:55.808408 25876 server_base.cc:1034] running on GCE node
I20260929 16:33:55.809267 25876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:55.810925 25876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:55.812150 25876 hybrid_clock.cc:648] HybridClock initialized: now 1790699635812100 us; error 38 us; skew 500 ppm
I20260929 16:33:55.812552 25876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:55.821095 25876 webserver.cc:460] Webserver started at http://127.5.232.190:36805/ using document root <none> and password file <none>
I20260929 16:33:55.821977 25876 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:55.822111 25876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:55.822484 25876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:55.825567 25876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a"
format_stamp: "Formatted at 2026-09-29 16:33:55 on dist-test-slave-dssx"
server_key: "f175981cdd9811e2775d162a42f87cc0"
server_key_iv: "a1fe1d282dbcf2b036ae83bdd7964b71"
server_key_version: "encryptionkey@0"
I20260929 16:33:55.826345 25876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a"
format_stamp: "Formatted at 2026-09-29 16:33:55 on dist-test-slave-dssx"
server_key: "f175981cdd9811e2775d162a42f87cc0"
server_key_iv: "a1fe1d282dbcf2b036ae83bdd7964b71"
server_key_version: "encryptionkey@0"
I20260929 16:33:55.831776 25876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:33:55.835470 25890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:55.836910 25876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 16:33:55.837071 25876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a"
format_stamp: "Formatted at 2026-09-29 16:33:55 on dist-test-slave-dssx"
server_key: "f175981cdd9811e2775d162a42f87cc0"
server_key_iv: "a1fe1d282dbcf2b036ae83bdd7964b71"
server_key_version: "encryptionkey@0"
I20260929 16:33:55.837286 25876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:55.860417 25876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:55.861277 25876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:55.861573 25876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:55.881009 25876 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:42413
I20260929 16:33:55.881029 25941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:42413 every 8 connection(s)
I20260929 16:33:55.882535 25876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:55.883819 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 25876
I20260929 16:33:55.883991 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:33:55.884543 6050 external_mini_cluster.cc:1428] Setting key db5fb236f7b23bc85d773c0068d256ea
I20260929 16:33:55.887542 25942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:55.898660 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:55.886595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.5.232.190:42413)
0929 16:33:55.887642 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:55.887657 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:55.887715 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:33:55.889469 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:55.889492 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:55.889965 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:55.890406 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:55.890995 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:55.892421 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:55.893951 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:55.894646 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:55.894858 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:55.896141 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:55.896150 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:55.896168 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:55.896783 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:55.896869 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:55.896891 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:55.897049 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:55.897303 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 16:33:55.897663 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":361,"threads_started":1}
I20260929 16:33:55.899132 25942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a: Bootstrap starting.
I20260929 16:33:55.902583 25942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:55.903652 25942 log.cc:824] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:55.906692 25942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a: No bootstrap required, opened a new log
I20260929 16:33:55.914776 25942 raft_consensus.cc:348] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42413 } }
I20260929 16:33:55.915263 25942 raft_consensus.cc:374] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:55.915380 25942 raft_consensus.cc:729] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85303a4fbf147f1b3ff55fbc7d95d6a, State: Initialized, Role: FOLLOWER
I20260929 16:33:55.915982 25942 consensus_queue.cc:260] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42413 } }
I20260929 16:33:55.916230 25942 raft_consensus.cc:388] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:55.916384 25942 raft_consensus.cc:482] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:55.916594 25942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:55.919235 25942 raft_consensus.cc:504] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42413 } }
I20260929 16:33:55.919787 25942 leader_election.cc:302] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f85303a4fbf147f1b3ff55fbc7d95d6a; no voters:
I20260929 16:33:55.920393 25942 leader_election.cc:288] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:55.920575 25947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:55.921324 25947 raft_consensus.cc:686] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [term 1 LEADER]: Becoming Leader. State: Replica: f85303a4fbf147f1b3ff55fbc7d95d6a, State: Running, Role: LEADER
I20260929 16:33:55.921870 25947 consensus_queue.cc:237] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42413 } }
I20260929 16:33:55.922561 25942 sys_catalog.cc:564] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:55.924732 25949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f85303a4fbf147f1b3ff55fbc7d95d6a. Latest consensus state: current_term: 1 leader_uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42413 } } }
I20260929 16:33:55.925087 25949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:55.925607 25948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85303a4fbf147f1b3ff55fbc7d95d6a" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 42413 } } }
I20260929 16:33:55.925931 25948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:55.926117 25954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:55.931711 25954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:33:55.944932 25954 catalog_manager.cc:1347] Generated new cluster ID: a326d54645ec44ab879d9c70aafe7715
I20260929 16:33:55.945068 25954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:33:55.957474 25954 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:33:55.959168 25954 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:33:55.970181 25954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a: Generated new TSK 0
I20260929 16:33:55.971109 25954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:33:55.986750 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.129:0
--local_ip_for_outbound_sockets=127.5.232.129
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42413
--builtin_ntp_servers=127.5.232.148:41429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:33:56.328485 25966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:56.328766 25966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:56.328915 25966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:56.336408 25966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:56.336508 25966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:56.336674 25966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.129
I20260929 16:33:56.346246 25966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.232.129
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.129
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:56.347576 25966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:56.349418 25966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:56.362946 25974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:56.362910 25972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:56.362881 25971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:56.366693 25966 server_base.cc:1034] running on GCE node
I20260929 16:33:56.367374 25966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:56.368312 25966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:56.369541 25966 hybrid_clock.cc:648] HybridClock initialized: now 1790699636369501 us; error 50 us; skew 500 ppm
I20260929 16:33:56.369867 25966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:56.372130 25966 webserver.cc:460] Webserver started at http://127.5.232.129:36089/ using document root <none> and password file <none>
I20260929 16:33:56.372838 25966 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:56.372974 25966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:56.373308 25966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:56.376240 25966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/data/instance:
uuid: "11301c2f30134f239c9a34a82db2caaf"
format_stamp: "Formatted at 2026-09-29 16:33:56 on dist-test-slave-dssx"
server_key: "0eb7b3d20d87cce6177c56d16252ad3f"
server_key_iv: "99957635750fdd88ca13ec00df941569"
server_key_version: "encryptionkey@0"
I20260929 16:33:56.377034 25966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance:
uuid: "11301c2f30134f239c9a34a82db2caaf"
format_stamp: "Formatted at 2026-09-29 16:33:56 on dist-test-slave-dssx"
server_key: "0eb7b3d20d87cce6177c56d16252ad3f"
server_key_iv: "99957635750fdd88ca13ec00df941569"
server_key_version: "encryptionkey@0"
I20260929 16:33:56.382220 25966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:33:56.385648 25980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:56.386911 25966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:33:56.387084 25966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/wal
uuid: "11301c2f30134f239c9a34a82db2caaf"
format_stamp: "Formatted at 2026-09-29 16:33:56 on dist-test-slave-dssx"
server_key: "0eb7b3d20d87cce6177c56d16252ad3f"
server_key_iv: "99957635750fdd88ca13ec00df941569"
server_key_version: "encryptionkey@0"
I20260929 16:33:56.387306 25966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:56.406041 25966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:56.406888 25966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:56.407196 25966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:56.408432 25966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:56.409976 25966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:56.410099 25966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:56.410226 25966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:56.410318 25966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:56.442958 25966 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.129:34537
I20260929 16:33:56.443012 26092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.129:34537 every 8 connection(s)
I20260929 16:33:56.444319 25966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/data/info.pb
I20260929 16:33:56.452389 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 25966
I20260929 16:33:56.452574 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/wal/instance
I20260929 16:33:56.453011 6050 external_mini_cluster.cc:1428] Setting key 249d99f827ade6cc3d567cfb48788715
I20260929 16:33:56.456014 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.130:0
--local_ip_for_outbound_sockets=127.5.232.130
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42413
--builtin_ntp_servers=127.5.232.148:41429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:33:56.457885 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:56.446872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.129:33969 (local address 127.5.232.190:42413)
0929 16:33:56.447347 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:56.447353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:56.448506 (+ 1153us) server_negotiation.cc:407] Connection header received
0929 16:33:56.450327 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:56.450331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:56.450399 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:56.450504 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:56.451603 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:56.452600 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:56.454774 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:56.455356 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:56.455500 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:56.456879 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:56.456888 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:56.456896 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:56.456921 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:56.456975 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:56.456993 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:56.457093 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:56.457307 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 16:33:56.457444 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20260929 16:33:56.458742 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:56.447482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42413 (local address 127.5.232.129:33969)
0929 16:33:56.448347 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:56.448383 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:56.450079 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:56.450705 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:56.450713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:56.450798 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:56.451377 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:56.451394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:56.452782 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:56.452788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:56.454615 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:56.454629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:56.455576 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:56.455581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:56.455776 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:56.456533 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:33:56.456535 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:56.456585 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:33:56.456670 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:56.457094 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:56.457102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:56.457108 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:33:56.457257 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 16:33:56.457633 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":263,"threads_started":1}
I20260929 16:33:56.460358 26093 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42413
I20260929 16:33:56.460683 26093 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:56.461665 26093 heartbeater.cc:511] Master 127.5.232.190:42413 requested a full tablet report, sending...
I20260929 16:33:56.463850 25907 ts_manager.cc:194] Registered new tserver with Master: 11301c2f30134f239c9a34a82db2caaf (127.5.232.129:34537)
I20260929 16:33:56.465611 25907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.129:33969
W20260929 16:33:56.789143 26097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:56.789517 26097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:56.789623 26097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:56.796792 26097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:56.796890 26097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:56.796994 26097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.130
I20260929 16:33:56.805938 26097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.232.130
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.130
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:56.807147 26097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:56.808880 26097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:56.821568 26102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:56.821545 26105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:56.821617 26103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:56.822692 26097 server_base.cc:1034] running on GCE node
I20260929 16:33:56.823294 26097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:56.824340 26097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:56.825564 26097 hybrid_clock.cc:648] HybridClock initialized: now 1790699636825519 us; error 59 us; skew 500 ppm
I20260929 16:33:56.825886 26097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:56.828140 26097 webserver.cc:460] Webserver started at http://127.5.232.130:44739/ using document root <none> and password file <none>
I20260929 16:33:56.828830 26097 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:56.828969 26097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:56.829288 26097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:56.832175 26097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/data/instance:
uuid: "3952c992a04840cea3d7d58d2179b2ee"
format_stamp: "Formatted at 2026-09-29 16:33:56 on dist-test-slave-dssx"
server_key: "48927949d321ddf5c9d4fef65a7c6a8f"
server_key_iv: "7d69d72e6960990d2c84c930aa5372b4"
server_key_version: "encryptionkey@0"
I20260929 16:33:56.832969 26097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance:
uuid: "3952c992a04840cea3d7d58d2179b2ee"
format_stamp: "Formatted at 2026-09-29 16:33:56 on dist-test-slave-dssx"
server_key: "48927949d321ddf5c9d4fef65a7c6a8f"
server_key_iv: "7d69d72e6960990d2c84c930aa5372b4"
server_key_version: "encryptionkey@0"
I20260929 16:33:56.838066 26097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:33:56.841511 26111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:56.842813 26097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:33:56.842996 26097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/wal
uuid: "3952c992a04840cea3d7d58d2179b2ee"
format_stamp: "Formatted at 2026-09-29 16:33:56 on dist-test-slave-dssx"
server_key: "48927949d321ddf5c9d4fef65a7c6a8f"
server_key_iv: "7d69d72e6960990d2c84c930aa5372b4"
server_key_version: "encryptionkey@0"
I20260929 16:33:56.843221 26097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:56.858181 26097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:56.859051 26097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:56.859375 26097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:56.860378 26097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:56.861929 26097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:56.862069 26097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:56.862213 26097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:56.862303 26097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:56.896152 26097 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.130:33925
I20260929 16:33:56.896183 26223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.130:33925 every 8 connection(s)
I20260929 16:33:56.897747 26097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/data/info.pb
I20260929 16:33:56.903255 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 26097
I20260929 16:33:56.903455 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/wal/instance
I20260929 16:33:56.904000 6050 external_mini_cluster.cc:1428] Setting key 62b85363f90bf7dfe3fed4dc705640a5
I20260929 16:33:56.907116 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.232.131:0
--local_ip_for_outbound_sockets=127.5.232.131
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42413
--builtin_ntp_servers=127.5.232.148:41429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:33:56.913069 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:56.900661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:35759 (local address 127.5.232.190:42413)
0929 16:33:56.900887 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:56.900894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:56.902529 (+ 1635us) server_negotiation.cc:407] Connection header received
0929 16:33:56.905037 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:56.905044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:56.905161 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:56.905275 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:56.906731 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:56.907572 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:56.910212 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:56.910587 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:56.910682 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:56.912337 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:56.912342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:56.912345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:56.912363 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:56.912395 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:56.912406 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:56.912489 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:56.912758 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:33:56.912841 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:33:56.914644 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:56.901177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42413 (local address 127.5.232.130:35759)
0929 16:33:56.902335 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:56.902393 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:56.904694 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:56.905591 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:56.905600 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:56.905700 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:56.906545 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:56.906572 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:56.907741 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:56.907750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:56.910059 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:56.910078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:56.910815 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:56.910824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:56.911162 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:56.911942 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:33:56.911948 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:56.912003 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:33:56.912118 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:56.912517 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:56.912529 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:56.912540 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:33:56.912725 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 16:33:56.913176 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":425,"threads_started":1}
I20260929 16:33:56.916429 26224 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42413
I20260929 16:33:56.916807 26224 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:56.917675 26224 heartbeater.cc:511] Master 127.5.232.190:42413 requested a full tablet report, sending...
I20260929 16:33:56.919224 25907 ts_manager.cc:194] Registered new tserver with Master: 3952c992a04840cea3d7d58d2179b2ee (127.5.232.130:33925)
I20260929 16:33:56.920073 25907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.130:35759
W20260929 16:33:57.247639 26227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:57.248005 26227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:57.248173 26227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:57.255738 26227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:57.255854 26227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:57.256036 26227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.232.131
I20260929 16:33:57.265177 26227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.232.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.232.131
--webserver_port=0
--tserver_master_addrs=127.5.232.190:42413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.232.131
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:57.266551 26227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:57.268338 26227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:57.281953 26235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:57.282073 26233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:57.282042 26232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:57.282948 26227 server_base.cc:1034] running on GCE node
I20260929 16:33:57.283645 26227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:57.284742 26227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:57.285969 26227 hybrid_clock.cc:648] HybridClock initialized: now 1790699637285897 us; error 71 us; skew 500 ppm
I20260929 16:33:57.286309 26227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:33:57.288628 26227 webserver.cc:460] Webserver started at http://127.5.232.131:39489/ using document root <none> and password file <none>
I20260929 16:33:57.289429 26227 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:57.289573 26227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:57.289904 26227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:57.293043 26227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/data/instance:
uuid: "d9c29196d2db4e26b147a49694cf98e7"
format_stamp: "Formatted at 2026-09-29 16:33:57 on dist-test-slave-dssx"
server_key: "cd56ed59af712d6fc5cd25903e726939"
server_key_iv: "a1a61c78835d5485849f000f3c7da5b1"
server_key_version: "encryptionkey@0"
I20260929 16:33:57.293885 26227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance:
uuid: "d9c29196d2db4e26b147a49694cf98e7"
format_stamp: "Formatted at 2026-09-29 16:33:57 on dist-test-slave-dssx"
server_key: "cd56ed59af712d6fc5cd25903e726939"
server_key_iv: "a1a61c78835d5485849f000f3c7da5b1"
server_key_version: "encryptionkey@0"
I20260929 16:33:57.299120 26227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:33:57.302886 26241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:57.304524 26227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:33:57.304713 26227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/wal
uuid: "d9c29196d2db4e26b147a49694cf98e7"
format_stamp: "Formatted at 2026-09-29 16:33:57 on dist-test-slave-dssx"
server_key: "cd56ed59af712d6fc5cd25903e726939"
server_key_iv: "a1a61c78835d5485849f000f3c7da5b1"
server_key_version: "encryptionkey@0"
I20260929 16:33:57.304955 26227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:57.316250 26227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:57.317109 26227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:57.317494 26227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:57.318514 26227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:33:57.320011 26227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:33:57.320106 26227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:57.320272 26227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:33:57.320361 26227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:57.354991 26227 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.131:45633
I20260929 16:33:57.355036 26353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.131:45633 every 8 connection(s)
I20260929 16:33:57.356384 26227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/data/info.pb
I20260929 16:33:57.366041 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 26227
I20260929 16:33:57.366230 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/wal/instance
I20260929 16:33:57.366732 6050 external_mini_cluster.cc:1428] Setting key e77cc773855b0745efe70fba14584313
I20260929 16:33:57.369603 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.359090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.131:56995 (local address 127.5.232.190:42413)
0929 16:33:57.359314 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.359318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.360681 (+ 1363us) server_negotiation.cc:407] Connection header received
0929 16:33:57.362538 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.362543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.362616 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.362708 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:57.363777 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.364508 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.366549 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.367058 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.367193 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.368751 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:57.368759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:57.368763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:57.368786 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:57.368823 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:57.368841 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:57.368932 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.369191 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.369272 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 16:33:57.370831 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.359608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:42413 (local address 127.5.232.131:56995)
0929 16:33:57.360520 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:57.360557 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:57.362278 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:57.362919 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:57.362927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:57.363038 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:33:57.363618 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.363635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.364636 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.364639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.366372 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.366390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.367292 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.367298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.367592 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.368357 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:33:57.368360 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:33:57.368412 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:33:57.368549 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:33:57.368958 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:33:57.368970 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:33:57.368979 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:33:57.369182 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 16:33:57.369676 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":296,"threads_started":1}
I20260929 16:33:57.372282 26354 heartbeater.cc:347] Connected to a master server at 127.5.232.190:42413
I20260929 16:33:57.372602 26354 heartbeater.cc:464] Registering TS with master...
I20260929 16:33:57.373435 26354 heartbeater.cc:511] Master 127.5.232.190:42413 requested a full tablet report, sending...
I20260929 16:33:57.374900 25907 ts_manager.cc:194] Registered new tserver with Master: d9c29196d2db4e26b147a49694cf98e7 (127.5.232.131:45633)
I20260929 16:33:57.375728 25907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.232.131:56995
I20260929 16:33:57.382644 6050 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:33:57.393165 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.386691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.5.232.190:42413)
0929 16:33:57.386942 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.386949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.387237 (+ 288us) server_negotiation.cc:407] Connection header received
0929 16:33:57.387387 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.387391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.387467 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.387555 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:57.388052 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.388753 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.390776 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.391204 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.391351 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.392450 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:57.392456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:57.392459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:57.392478 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:57.392511 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:57.392524 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:57.392624 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.392766 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.392862 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:33:57.397456 25907 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:33:57.405610 25907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:33:57.440585 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.418062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.130:33925 (local address 127.0.0.1:35602)
0929 16:33:57.419532 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:57.419585 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:57.419782 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:57.432191 (+ 12409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:57.432199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:57.432231 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:57.432713 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.432731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.434939 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.434944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.437534 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.437549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.439553 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.439560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.439792 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.439857 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 16:33:57.439968 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:33:57.440161 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"thread_start_us":13207,"threads_started":1}
I20260929 16:33:57.443174 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.420314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.5.232.131:45633)
0929 16:33:57.431101 (+ 10787us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.431114 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.431698 (+ 584us) server_negotiation.cc:407] Connection header received
0929 16:33:57.432012 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.432019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.432264 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.432447 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:57.433430 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.436157 (+ 2727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.438791 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.440263 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.440371 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.442040 (+ 1669us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.442629 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.442814 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10585,"thread_start_us":10426,"threads_started":1}
I20260929 16:33:57.444229 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.418531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:45633 (local address 127.0.0.1:49228)
0929 16:33:57.431581 (+ 13050us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:57.431602 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:57.431808 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:57.432760 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:57.432765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:57.432796 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:57.433201 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.433213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.436348 (+ 3135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.436353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.438568 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.438585 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.442227 (+ 3642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.442234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.442409 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.442440 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:33:57.443659 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0929 16:33:57.443823 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12900,"mutex_wait_us":1,"thread_start_us":2038,"threads_started":1}
I20260929 16:33:57.449878 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.418239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.5.232.130:33925)
0929 16:33:57.431418 (+ 13179us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.431430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.431464 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:33:57.431603 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.431614 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.431927 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.432951 (+ 1024us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:57.433198 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.434775 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.437793 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.439319 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.441348 (+ 2029us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.441661 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.441749 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.441929 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13011,"thread_start_us":12878,"threads_started":1}
I20260929 16:33:57.448794 26289 tablet_service.cc:1467] Processing CreateTablet for tablet e4df40e2877a4be4afd1cb45762e3ca7 (DEFAULT_TABLE table=test-table [id=268033ce721745408489c6a361a4b9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:57.451952 26289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4df40e2877a4be4afd1cb45762e3ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:57.453652 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.431761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34537 (local address 127.0.0.1:43216)
0929 16:33:57.440970 (+ 9209us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:57.440993 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:57.441095 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:57.445665 (+ 4570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:57.445674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:57.445717 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:57.445967 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.445982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.449624 (+ 3642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.449633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.451379 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.451394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.452798 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.452804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.452999 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.453043 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 16:33:57.453153 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:33:57.453303 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9077,"thread_start_us":17516,"threads_started":1}
I20260929 16:33:57.453732 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.431994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.5.232.129:34537)
0929 16:33:57.444504 (+ 12510us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.444517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.444559 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:33:57.444687 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.444699 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.445060 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.446198 (+ 1138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:57.446580 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.448951 (+ 2371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.451555 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.452618 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.452786 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.453066 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.453202 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.453358 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12258,"thread_start_us":13267,"threads_started":1}
I20260929 16:33:57.454270 26159 tablet_service.cc:1467] Processing CreateTablet for tablet e4df40e2877a4be4afd1cb45762e3ca7 (DEFAULT_TABLE table=test-table [id=268033ce721745408489c6a361a4b9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:57.455794 26159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4df40e2877a4be4afd1cb45762e3ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:57.456966 26028 tablet_service.cc:1467] Processing CreateTablet for tablet e4df40e2877a4be4afd1cb45762e3ca7 (DEFAULT_TABLE table=test-table [id=268033ce721745408489c6a361a4b9f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:33:57.458215 26028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4df40e2877a4be4afd1cb45762e3ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:33:57.467203 26372 tablet_bootstrap.cc:492] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: Bootstrap starting.
I20260929 16:33:57.468653 26093 heartbeater.cc:503] Master 127.5.232.190:42413 was elected leader, sending a full tablet report...
I20260929 16:33:57.471583 26374 tablet_bootstrap.cc:492] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: Bootstrap starting.
I20260929 16:33:57.471702 26372 tablet_bootstrap.cc:654] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:57.472719 26373 tablet_bootstrap.cc:492] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: Bootstrap starting.
I20260929 16:33:57.473773 26372 log.cc:824] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:57.476199 26374 tablet_bootstrap.cc:654] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:57.476547 26373 tablet_bootstrap.cc:654] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:57.477530 26374 log.cc:824] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:57.477629 26373 log.cc:824] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:57.477797 26372 tablet_bootstrap.cc:492] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: No bootstrap required, opened a new log
I20260929 16:33:57.478111 26372 ts_tablet_manager.cc:1397] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260929 16:33:57.480948 26374 tablet_bootstrap.cc:492] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: No bootstrap required, opened a new log
I20260929 16:33:57.481236 26374 ts_tablet_manager.cc:1397] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:33:57.483843 26373 tablet_bootstrap.cc:492] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: No bootstrap required, opened a new log
I20260929 16:33:57.484138 26373 ts_tablet_manager.cc:1397] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260929 16:33:57.488909 26372 raft_consensus.cc:348] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } }
I20260929 16:33:57.489516 26372 raft_consensus.cc:374] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:57.489676 26372 raft_consensus.cc:729] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c29196d2db4e26b147a49694cf98e7, State: Initialized, Role: FOLLOWER
I20260929 16:33:57.490419 26372 consensus_queue.cc:260] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } }
I20260929 16:33:57.492388 26373 raft_consensus.cc:348] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } }
I20260929 16:33:57.492797 26373 raft_consensus.cc:374] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:57.492926 26373 raft_consensus.cc:729] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11301c2f30134f239c9a34a82db2caaf, State: Initialized, Role: FOLLOWER
I20260929 16:33:57.493314 26374 raft_consensus.cc:348] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } }
I20260929 16:33:57.493680 26373 consensus_queue.cc:260] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } }
I20260929 16:33:57.493896 26374 raft_consensus.cc:374] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:57.494045 26374 raft_consensus.cc:729] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3952c992a04840cea3d7d58d2179b2ee, State: Initialized, Role: FOLLOWER
I20260929 16:33:57.494169 26354 heartbeater.cc:503] Master 127.5.232.190:42413 was elected leader, sending a full tablet report...
I20260929 16:33:57.495657 26373 ts_tablet_manager.cc:1428] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 16:33:57.496382 26372 ts_tablet_manager.cc:1428] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: Time spent starting tablet: real 0.018s user 0.008s sys 0.008s
I20260929 16:33:57.494791 26374 consensus_queue.cc:260] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } }
I20260929 16:33:57.497608 26224 heartbeater.cc:503] Master 127.5.232.190:42413 was elected leader, sending a full tablet report...
I20260929 16:33:57.498101 26374 ts_tablet_manager.cc:1428] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260929 16:33:57.498481 26380 raft_consensus.cc:482] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:33:57.498796 26380 raft_consensus.cc:504] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } }
I20260929 16:33:57.500774 26380 leader_election.cc:288] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11301c2f30134f239c9a34a82db2caaf (127.5.232.129:34537), d9c29196d2db4e26b147a49694cf98e7 (127.5.232.131:45633)
I20260929 16:33:57.508787 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.501424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:32883 (local address 127.5.232.129:34537)
0929 16:33:57.502067 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.502072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.502205 (+ 133us) server_negotiation.cc:407] Connection header received
0929 16:33:57.502582 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.502590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.502692 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.502770 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:57.503859 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.504956 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.506923 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.507715 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.507937 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.508009 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.508390 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.508524 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491}
I20260929 16:33:57.508857 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.501105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.129:34537 (local address 127.5.232.130:32883)
0929 16:33:57.502089 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:57.502113 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:57.502349 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:57.503011 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:57.503018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:57.503052 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:57.503607 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.503623 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.505132 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.505140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.506709 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.506721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.507945 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.507951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.508186 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.508248 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 16:33:57.508358 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:33:57.508507 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":55,"thread_start_us":602,"threads_started":1}
I20260929 16:33:57.510020 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4df40e2877a4be4afd1cb45762e3ca7" candidate_uuid: "3952c992a04840cea3d7d58d2179b2ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11301c2f30134f239c9a34a82db2caaf" is_pre_election: true
I20260929 16:33:57.510576 26048 raft_consensus.cc:2454] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3952c992a04840cea3d7d58d2179b2ee in term 0.
I20260929 16:33:57.510747 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.502371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.232.130:52767 (local address 127.5.232.131:45633)
0929 16:33:57.502539 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.502546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.505161 (+ 2615us) server_negotiation.cc:407] Connection header received
0929 16:33:57.505421 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.505426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.505496 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.505609 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:33:57.506195 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.506885 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.508552 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.509860 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.510049 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.510117 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.510382 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.510506 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:33:57.510874 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.501831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.131:45633 (local address 127.5.232.130:52767)
0929 16:33:57.505074 (+ 3243us) negotiation.cc:107] Waiting for socket to connect
0929 16:33:57.505093 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:33:57.505268 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:33:57.505630 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:33:57.505634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:33:57.505661 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:33:57.506024 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.506035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.507035 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.507041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.508348 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:33:57.508359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.510049 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.510056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:33:57.510222 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.510245 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:33:57.510356 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:33:57.510504 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3095,"mutex_wait_us":3,"thread_start_us":2906,"threads_started":1}
I20260929 16:33:57.511339 26112 leader_election.cc:302] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11301c2f30134f239c9a34a82db2caaf, 3952c992a04840cea3d7d58d2179b2ee; no voters:
I20260929 16:33:57.511756 26380 raft_consensus.cc:2781] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:33:57.511891 26380 raft_consensus.cc:482] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:33:57.512022 26380 raft_consensus.cc:3037] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:57.512138 26309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4df40e2877a4be4afd1cb45762e3ca7" candidate_uuid: "3952c992a04840cea3d7d58d2179b2ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c29196d2db4e26b147a49694cf98e7" is_pre_election: true
I20260929 16:33:57.512622 26309 raft_consensus.cc:2454] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3952c992a04840cea3d7d58d2179b2ee in term 0.
I20260929 16:33:57.514581 26380 raft_consensus.cc:504] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } }
I20260929 16:33:57.515273 26380 leader_election.cc:288] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [CANDIDATE]: Term 1 election: Requested vote from peers 11301c2f30134f239c9a34a82db2caaf (127.5.232.129:34537), d9c29196d2db4e26b147a49694cf98e7 (127.5.232.131:45633)
I20260929 16:33:57.515785 26309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4df40e2877a4be4afd1cb45762e3ca7" candidate_uuid: "3952c992a04840cea3d7d58d2179b2ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c29196d2db4e26b147a49694cf98e7"
I20260929 16:33:57.515786 26048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4df40e2877a4be4afd1cb45762e3ca7" candidate_uuid: "3952c992a04840cea3d7d58d2179b2ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11301c2f30134f239c9a34a82db2caaf"
I20260929 16:33:57.516008 26309 raft_consensus.cc:3037] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:57.516009 26048 raft_consensus.cc:3037] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:57.519332 26048 raft_consensus.cc:2454] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3952c992a04840cea3d7d58d2179b2ee in term 1.
I20260929 16:33:57.519332 26309 raft_consensus.cc:2454] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3952c992a04840cea3d7d58d2179b2ee in term 1.
I20260929 16:33:57.519840 26114 leader_election.cc:302] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11301c2f30134f239c9a34a82db2caaf, 3952c992a04840cea3d7d58d2179b2ee, d9c29196d2db4e26b147a49694cf98e7; no voters:
I20260929 16:33:57.520210 26380 raft_consensus.cc:2781] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:57.520680 26380 raft_consensus.cc:686] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 1 LEADER]: Becoming Leader. State: Replica: 3952c992a04840cea3d7d58d2179b2ee, State: Running, Role: LEADER
I20260929 16:33:57.521179 26380 consensus_queue.cc:237] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } }
I20260929 16:33:57.525964 25906 catalog_manager.cc:5495] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 3952c992a04840cea3d7d58d2179b2ee (127.5.232.130). New cstate: current_term: 1 leader_uuid: "3952c992a04840cea3d7d58d2179b2ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3952c992a04840cea3d7d58d2179b2ee" member_type: VOTER last_known_addr { host: "127.5.232.130" port: 33925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 } health_report { overall_health: UNKNOWN } } }
I20260929 16:33:57.556044 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.551049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.5.232.130:33925)
0929 16:33:57.551231 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.551236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.551309 (+ 73us) server_negotiation.cc:407] Connection header received
0929 16:33:57.551495 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.551499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.551589 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.551725 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:33:57.552041 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.552720 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.554032 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.554510 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.554615 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.555064 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:33:57.555569 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:33:57.555690 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.555756 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.555821 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:33:57.565337 26048 raft_consensus.cc:1261] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [term 1 FOLLOWER]: Refusing update from remote peer 3952c992a04840cea3d7d58d2179b2ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:57.565341 26309 raft_consensus.cc:1261] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [term 1 FOLLOWER]: Refusing update from remote peer 3952c992a04840cea3d7d58d2179b2ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:33:57.566594 26383 consensus_queue.cc:1035] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c29196d2db4e26b147a49694cf98e7" member_type: VOTER last_known_addr { host: "127.5.232.131" port: 45633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:57.566987 26380 consensus_queue.cc:1035] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "11301c2f30134f239c9a34a82db2caaf" member_type: VOTER last_known_addr { host: "127.5.232.129" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:33:57.579383 26387 mvcc.cc:205] Tried to move back new op lower bound from 7334705715455705088 to 7334705715291435008. Current Snapshot: MvccSnapshot[applied={T|T < 7334705715455705088}]
I20260929 16:33:57.579485 26386 mvcc.cc:205] Tried to move back new op lower bound from 7334705715455705088 to 7334705715291435008. Current Snapshot: MvccSnapshot[applied={T|T < 7334705715455705088}]
I20260929 16:33:57.594317 25906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:57.594630 25906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:33:57.598547 25906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f85303a4fbf147f1b3ff55fbc7d95d6a: Sending DeleteTablet for 3 replicas of tablet e4df40e2877a4be4afd1cb45762e3ca7
I20260929 16:33:57.599798 26159 tablet_service.cc:1514] Processing DeleteTablet for tablet e4df40e2877a4be4afd1cb45762e3ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:57 UTC) from {username='slave'} at 127.0.0.1:35602
I20260929 16:33:57.599859 26028 tablet_service.cc:1514] Processing DeleteTablet for tablet e4df40e2877a4be4afd1cb45762e3ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:57 UTC) from {username='slave'} at 127.0.0.1:43216
I20260929 16:33:57.600311 26289 tablet_service.cc:1514] Processing DeleteTablet for tablet e4df40e2877a4be4afd1cb45762e3ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:33:57 UTC) from {username='slave'} at 127.0.0.1:49228
I20260929 16:33:57.602181 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 25966
I20260929 16:33:57.616863 26396 tablet_replica.cc:330] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: stopping tablet replica
I20260929 16:33:57.617486 26396 raft_consensus.cc:2229] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:57.618139 26396 raft_consensus.cc:2258] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:57.619060 26395 tablet_replica.cc:330] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: stopping tablet replica
I20260929 16:33:57.619791 26395 raft_consensus.cc:2229] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:33:57.620684 26395 raft_consensus.cc:2258] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:57.623319 26395 ts_tablet_manager.cc:1905] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:57.626008 26397 tablet_replica.cc:330] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: stopping tablet replica
I20260929 16:33:57.626525 26397 raft_consensus.cc:2229] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:33:57.627137 26397 raft_consensus.cc:2258] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:33:57.628211 26396 ts_tablet_manager.cc:1905] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:57.629281 26397 ts_tablet_manager.cc:1905] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:33:57.631312 26395 ts_tablet_manager.cc:1918] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:57.631567 26395 log.cc:1196] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-1/wal/wals/e4df40e2877a4be4afd1cb45762e3ca7
I20260929 16:33:57.632243 26395 ts_tablet_manager.cc:1939] T e4df40e2877a4be4afd1cb45762e3ca7 P 3952c992a04840cea3d7d58d2179b2ee: Deleting consensus metadata
I20260929 16:33:57.634342 25891 catalog_manager.cc:4841] TS 3952c992a04840cea3d7d58d2179b2ee (127.5.232.130:33925): tablet e4df40e2877a4be4afd1cb45762e3ca7 (table test-table [id=268033ce721745408489c6a361a4b9f1]) successfully deleted
I20260929 16:33:57.635550 26396 ts_tablet_manager.cc:1918] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:57.635743 26396 log.cc:1196] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-0/wal/wals/e4df40e2877a4be4afd1cb45762e3ca7
I20260929 16:33:57.635825 26397 ts_tablet_manager.cc:1918] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:33:57.636037 26397 log.cc:1196] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: Deleting WAL directory at /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699314866137-6050-0/minicluster-data/ts-2/wal/wals/e4df40e2877a4be4afd1cb45762e3ca7
I20260929 16:33:57.636289 26396 ts_tablet_manager.cc:1939] T e4df40e2877a4be4afd1cb45762e3ca7 P 11301c2f30134f239c9a34a82db2caaf: Deleting consensus metadata
I20260929 16:33:57.636623 26397 ts_tablet_manager.cc:1939] T e4df40e2877a4be4afd1cb45762e3ca7 P d9c29196d2db4e26b147a49694cf98e7: Deleting consensus metadata
I20260929 16:33:57.638329 25891 catalog_manager.cc:4841] TS 11301c2f30134f239c9a34a82db2caaf (127.5.232.129:34537): tablet e4df40e2877a4be4afd1cb45762e3ca7 (table test-table [id=268033ce721745408489c6a361a4b9f1]) successfully deleted
I20260929 16:33:57.638494 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.603810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.5.232.129:34537)
0929 16:33:57.604036 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.604046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.621013 (+ 16967us) server_negotiation.cc:407] Connection header received
0929 16:33:57.621578 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.621586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.621712 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.621795 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:57.622474 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.623551 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.626087 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.626776 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.626888 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.636477 (+ 9589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:57.636495 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:57.636515 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:57.637441 (+ 926us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:57.637518 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:57.637554 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:57.637658 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.637959 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.638109 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:33:57.638440 25893 catalog_manager.cc:4841] TS d9c29196d2db4e26b147a49694cf98e7 (127.5.232.131:45633): tablet e4df40e2877a4be4afd1cb45762e3ca7 (table test-table [id=268033ce721745408489c6a361a4b9f1]) successfully deleted
I20260929 16:33:57.639688 26088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:57.734839 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 25966
I20260929 16:33:57.754200 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26097
I20260929 16:33:57.762166 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.755056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.5.232.130:33925)
0929 16:33:57.755242 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.755247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.755336 (+ 89us) server_negotiation.cc:407] Connection header received
0929 16:33:57.755533 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.755537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.755628 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.755713 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:57.756200 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.756994 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.758854 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.759296 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.759394 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.760624 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:57.760633 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:57.760649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:57.761275 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:57.761321 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:57.761341 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:57.761522 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.761669 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.761761 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:33:57.762897 26219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:57.878484 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 26097
I20260929 16:33:57.903223 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26227
I20260929 16:33:57.910857 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:57.904050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.5.232.131:45633)
0929 16:33:57.904226 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:57.904230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:57.904249 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:33:57.904442 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:57.904446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:57.904534 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:57.904620 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:57.905088 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.905921 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.907739 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:57.908114 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:57.908259 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:57.909475 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:57.909517 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:57.909538 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:33:57.910106 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:57.910151 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:33:57.910169 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:57.910257 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:57.910420 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 16:33:57.910520 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:33:57.911576 26349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:58.002163 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 26227
I20260929 16:33:58.019109 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 25876
I20260929 16:33:58.020262 25937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:33:58.130467 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 25876
2026-09-29T16:33:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2784 ms)
[----------] 6 tests from EncryptionPolicyTest (22245 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:33:58 dist-test-slave-dssx krb5kdc[26405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:33:58 dist-test-slave-dssx krb5kdc[26405](info): set up 2 sockets
Sep 29 16:33:58 dist-test-slave-dssx krb5kdc[26405](info): commencing operation
krb5kdc: starting...
W20260929 16:33:59.245951 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:33:59 dist-test-slave-dssx krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:33:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:33:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:33:59.501127 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:33:59.839254 26418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:33:59.839604 26418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:33:59.839771 26418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:33:59.847080 26418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:33:59.847251 26418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:33:59.847467 26418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:33:59.847604 26418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:33:59.847692 26418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:33:59.858204 26418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:33:59.859561 26418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:33:59.860991 26418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:33:59.873000 26423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:59.873000 26426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:33:59.872938 26424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:33:59.873788 26418 server_base.cc:1034] running on GCE node
I20260929 16:33:59.874547 26418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:33:59.875895 26418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:33:59.877103 26418 hybrid_clock.cc:648] HybridClock initialized: now 1790699639877046 us; error 53 us; skew 500 ppm
I20260929 16:33:59.877538 26418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:33:59 dist-test-slave-dssx krb5kdc[26405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:33:59.880677 26418 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:33:59.882280 26418 webserver.cc:460] Webserver started at http://127.0.0.1:41297/ using document root <none> and password file <none>
I20260929 16:33:59.882956 26418 fs_manager.cc:359] Metadata directory not provided
I20260929 16:33:59.883059 26418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:33:59.883303 26418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:33:59.886369 26418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "97d22d89384444d5926cb4786cf8ed8b"
format_stamp: "Formatted at 2026-09-29 16:33:59 on dist-test-slave-dssx"
server_key: "0898e860ad1118ce5b97ecb23b0f5544"
server_key_iv: "3fe8fee081e0151ef7760ed21a75e26b"
server_key_version: "encryptionkey@0"
I20260929 16:33:59.887120 26418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "97d22d89384444d5926cb4786cf8ed8b"
format_stamp: "Formatted at 2026-09-29 16:33:59 on dist-test-slave-dssx"
server_key: "0898e860ad1118ce5b97ecb23b0f5544"
server_key_iv: "3fe8fee081e0151ef7760ed21a75e26b"
server_key_version: "encryptionkey@0"
I20260929 16:33:59.892407 26418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:33:59.895932 26433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:33:59.897325 26418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:33:59.897604 26418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "97d22d89384444d5926cb4786cf8ed8b"
format_stamp: "Formatted at 2026-09-29 16:33:59 on dist-test-slave-dssx"
server_key: "0898e860ad1118ce5b97ecb23b0f5544"
server_key_iv: "3fe8fee081e0151ef7760ed21a75e26b"
server_key_version: "encryptionkey@0"
I20260929 16:33:59.897814 26418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:33:59.910646 26418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:33:59.914705 26418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:33:59.914988 26418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:33:59.933472 26418 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38359
I20260929 16:33:59.933521 26484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38359 every 8 connection(s)
I20260929 16:33:59.934865 26418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:33:59.935344 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 26418
I20260929 16:33:59.935565 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:33:59.936379 6050 external_mini_cluster.cc:1428] Setting key 22b2c24a873b32e471bdc69811257f6e
I20260929 16:33:59.939738 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:33:59 dist-test-slave-dssx krb5kdc[26405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:33:59.952420 26485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b: Bootstrap starting.
I20260929 16:33:59.956802 26485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b: Neither blocks nor log segments found. Creating new log.
I20260929 16:33:59.958163 26485 log.cc:824] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:33:59.961546 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:59.938531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60125 (local address 127.0.0.1:38359)
0929 16:33:59.939381 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:59.939398 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:59.939447 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:33:59.941854 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:59.941881 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:59.942361 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:59.942834 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:59.944133 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:59.946261 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:59.947478 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:59.948490 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:59.948679 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:59.952705 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:59.952716 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:59.952738 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:59.952794 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:59.956667 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:59.957511 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:59.957520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:59.957536 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:59.957754 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:59.958246 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:59.958251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:59.958255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:59.958992 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:59.959242 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:59.959856 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 16:33:59.960190 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":355,"threads_started":1}
I20260929 16:33:59.961681 26485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b: No bootstrap required, opened a new log
I20260929 16:33:59.965617 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:33:59.970306 26485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d22d89384444d5926cb4786cf8ed8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 16:33:59.970791 26485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:33:59.970912 26485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d22d89384444d5926cb4786cf8ed8b, State: Initialized, Role: FOLLOWER
I20260929 16:33:59.971642 26485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d22d89384444d5926cb4786cf8ed8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 16:33:59.971887 26485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:33:59.972038 26485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:33:59.972219 26485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:33:59.975061 26485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d22d89384444d5926cb4786cf8ed8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 16:33:59.975649 26485 leader_election.cc:302] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97d22d89384444d5926cb4786cf8ed8b; no voters:
I20260929 16:33:59.976322 26485 leader_election.cc:288] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:33:59.976455 26495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:33:59.978425 26485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:33:59.981644 26495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [term 1 LEADER]: Becoming Leader. State: Replica: 97d22d89384444d5926cb4786cf8ed8b, State: Running, Role: LEADER
I20260929 16:33:59.982322 26495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d22d89384444d5926cb4786cf8ed8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 16:33:59.986161 26496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97d22d89384444d5926cb4786cf8ed8b. Latest consensus state: current_term: 1 leader_uuid: "97d22d89384444d5926cb4786cf8ed8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d22d89384444d5926cb4786cf8ed8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } } }
I20260929 16:33:59.986662 26496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:59.986899 26497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97d22d89384444d5926cb4786cf8ed8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97d22d89384444d5926cb4786cf8ed8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } } }
I20260929 16:33:59.987294 26497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b [sys.catalog]: This master's current role is: LEADER
I20260929 16:33:59.988974 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:33:59.970451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51613 (local address 127.0.0.1:38359)
0929 16:33:59.970648 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:33:59.970656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:33:59.971073 (+ 417us) server_negotiation.cc:407] Connection header received
0929 16:33:59.971241 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:33:59.971247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:33:59.971327 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:33:59.971437 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:33:59.972937 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:59.974159 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:59.975456 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:33:59.976154 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:33:59.976316 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:33:59.980191 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:33:59.980201 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:33:59.980210 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:33:59.980272 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:33:59.984412 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:59.985488 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:59.985493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:59.985497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:59.985718 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:33:59.986321 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:33:59.986328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:33:59.986334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:33:59.986763 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:33:59.987123 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0929 16:33:59.988346 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0929 16:33:59.988534 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:33:59.990620 26505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:33:59.996047 26505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:00.004773 26505 catalog_manager.cc:1347] Generated new cluster ID: 54540000a88e49d0b4ce425fa72ef677
I20260929 16:34:00.004896 26505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:00.031709 26505 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:00.033084 26505 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:00.050462 26505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97d22d89384444d5926cb4786cf8ed8b: Generated new TSK 0
I20260929 16:34:00.051371 26505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:00.061307 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26418
I20260929 16:34:00.063409 26480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:00.157955 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 26418
W20260929 16:34:00.175453 26403 connection.cc:535] client connection to 127.0.0.1:38359 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38359): Connection reset by peer (error 104)
2026-09-29T16:34:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2017 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:00 dist-test-slave-dssx krb5kdc[26517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:00 dist-test-slave-dssx krb5kdc[26517](info): set up 2 sockets
Sep 29 16:34:00 dist-test-slave-dssx krb5kdc[26517](info): commencing operation
krb5kdc: starting...
W20260929 16:34:01.253734 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:01 dist-test-slave-dssx krb5kdc[26517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:01.500361 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:01.824122 26530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:01.824410 26530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:01.824600 26530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:01.831753 26530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:01.831863 26530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:01.831962 26530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:01.832062 26530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:01.832141 26530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:01.841864 26530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:01.843179 26530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:01.844480 26530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:01.856125 26535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:01.856144 26538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:01.856056 26536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:01.856819 26530 server_base.cc:1034] running on GCE node
I20260929 16:34:01.857616 26530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:01.858978 26530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:01.860170 26530 hybrid_clock.cc:648] HybridClock initialized: now 1790699641860124 us; error 34 us; skew 500 ppm
I20260929 16:34:01.860504 26530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:01 dist-test-slave-dssx krb5kdc[26517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:01.866901 26530 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:01.868363 26530 webserver.cc:460] Webserver started at http://127.0.0.1:40077/ using document root <none> and password file <none>
I20260929 16:34:01.869029 26530 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:01.869129 26530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:01.869447 26530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:01.872390 26530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "45aebc01638a40d79ad7e7766bcd7434"
format_stamp: "Formatted at 2026-09-29 16:34:01 on dist-test-slave-dssx"
server_key: "dc8dccbefe41d29c619c586a1dc7e4c2"
server_key_iv: "3cbfe740aa377e8f2250e627eb1bb426"
server_key_version: "encryptionkey@0"
I20260929 16:34:01.873095 26530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "45aebc01638a40d79ad7e7766bcd7434"
format_stamp: "Formatted at 2026-09-29 16:34:01 on dist-test-slave-dssx"
server_key: "dc8dccbefe41d29c619c586a1dc7e4c2"
server_key_iv: "3cbfe740aa377e8f2250e627eb1bb426"
server_key_version: "encryptionkey@0"
I20260929 16:34:01.878652 26530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:34:01.882201 26545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:01.883601 26530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:01.883800 26530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "45aebc01638a40d79ad7e7766bcd7434"
format_stamp: "Formatted at 2026-09-29 16:34:01 on dist-test-slave-dssx"
server_key: "dc8dccbefe41d29c619c586a1dc7e4c2"
server_key_iv: "3cbfe740aa377e8f2250e627eb1bb426"
server_key_version: "encryptionkey@0"
I20260929 16:34:01.883996 26530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:01.907977 26530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:01.912106 26530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:01.912395 26530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:01.931085 26530 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40245
I20260929 16:34:01.931125 26596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40245 every 8 connection(s)
I20260929 16:34:01.932492 26530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:01.936275 26597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:01.942619 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 26530
I20260929 16:34:01.942821 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:01.943367 6050 external_mini_cluster.cc:1428] Setting key f6a7e694d46bf8b64bb6724037edcee8
I20260929 16:34:01.947372 26597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434: Bootstrap starting.
I20260929 16:34:01.951005 26597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:01.952334 26597 log.cc:824] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:01.955471 26597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434: No bootstrap required, opened a new log
I20260929 16:34:01.957257 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:01.945218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38315 (local address 127.0.0.1:40245)
0929 16:34:01.946057 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:01.946069 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:01.946114 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:34:01.947825 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:01.947847 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:01.948252 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:01.948584 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:01.949226 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:01.950604 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:01.952640 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:01.953439 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:01.953690 (+ 251us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:01.954970 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:01.954977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:01.954993 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:01.955544 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:01.955657 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:01.955684 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:01.955841 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:01.956145 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:34:01.956310 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":423,"threads_started":1}
I20260929 16:34:01.961047 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:01.963563 26597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aebc01638a40d79ad7e7766bcd7434" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } }
I20260929 16:34:01.963956 26597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:01.964073 26597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45aebc01638a40d79ad7e7766bcd7434, State: Initialized, Role: FOLLOWER
I20260929 16:34:01.964711 26597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aebc01638a40d79ad7e7766bcd7434" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } }
I20260929 16:34:01.964938 26597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:01.965091 26597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:01.965292 26597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:01.968204 26597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aebc01638a40d79ad7e7766bcd7434" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } }
I20260929 16:34:01.968789 26597 leader_election.cc:302] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45aebc01638a40d79ad7e7766bcd7434; no voters:
I20260929 16:34:01.969365 26597 leader_election.cc:288] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:01.969553 26607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:01.971448 26597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:01.972554 26607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [term 1 LEADER]: Becoming Leader. State: Replica: 45aebc01638a40d79ad7e7766bcd7434, State: Running, Role: LEADER
I20260929 16:34:01.973620 26607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aebc01638a40d79ad7e7766bcd7434" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } }
I20260929 16:34:01.974236 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:01.965405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43523 (local address 127.0.0.1:40245)
0929 16:34:01.965569 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:01.965574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:01.966240 (+ 666us) server_negotiation.cc:407] Connection header received
0929 16:34:01.966472 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:01.966478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:01.966564 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:01.966664 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:01.967144 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:01.968249 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:01.969989 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:01.971008 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:01.971166 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:01.973065 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:01.973071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:01.973075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:01.973097 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:01.973201 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:01.973224 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:01.973343 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:01.973657 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:34:01.973723 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:34:01.976655 26608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45aebc01638a40d79ad7e7766bcd7434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aebc01638a40d79ad7e7766bcd7434" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } } }
I20260929 16:34:01.976975 26608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:01.977897 26614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:01.977770 26609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45aebc01638a40d79ad7e7766bcd7434. Latest consensus state: current_term: 1 leader_uuid: "45aebc01638a40d79ad7e7766bcd7434" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aebc01638a40d79ad7e7766bcd7434" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40245 } } }
I20260929 16:34:01.978142 26609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:01.984402 26614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:01.994158 26614 catalog_manager.cc:1347] Generated new cluster ID: 0331dec5e7fd4512914e14db18f00ab9
I20260929 16:34:01.994278 26614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:02.003875 26614 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:02.005573 26614 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:02.016810 26614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45aebc01638a40d79ad7e7766bcd7434: Generated new TSK 0
I20260929 16:34:02.017830 26614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:02.027804 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26530
W20260929 16:34:02.029273 26592 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38315
W20260929 16:34:02.030122 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26530: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:02.030323 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 26530
W20260929 16:34:02.047334 26515 connection.cc:535] client connection to 127.0.0.1:40245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40245): Connection reset by peer (error 104)
2026-09-29T16:34:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1871 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:02 dist-test-slave-dssx krb5kdc[26628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:02 dist-test-slave-dssx krb5kdc[26628](info): set up 2 sockets
Sep 29 16:34:02 dist-test-slave-dssx krb5kdc[26628](info): commencing operation
krb5kdc: starting...
W20260929 16:34:03.129712 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:03 dist-test-slave-dssx krb5kdc[26628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:03.385773 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:03.734395 26641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:03.734695 26641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:03.734799 26641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:03.742120 26641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:03.742234 26641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:03.742287 26641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:03.742347 26641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:03.742391 26641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:03.752271 26641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:03.753582 26641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:03.754837 26641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:03.766517 26649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:03.766551 26647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:03.766588 26646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:03.767283 26641 server_base.cc:1034] running on GCE node
I20260929 16:34:03.768034 26641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:03.769490 26641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:03.770761 26641 hybrid_clock.cc:648] HybridClock initialized: now 1790699643770690 us; error 50 us; skew 500 ppm
I20260929 16:34:03.771140 26641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:03 dist-test-slave-dssx krb5kdc[26628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:03.778494 26641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:03.779965 26641 webserver.cc:460] Webserver started at http://127.0.0.1:41999/ using document root <none> and password file <none>
I20260929 16:34:03.780660 26641 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:03.780789 26641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:03.781083 26641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:03.784101 26641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "c0fd7ec8a3e243888fdf2362e406e972"
format_stamp: "Formatted at 2026-09-29 16:34:03 on dist-test-slave-dssx"
server_key: "5f8ee1f4f13574eb06c6274d580def1b"
server_key_iv: "0eb6df6a09900263b6ea3d0e06432631"
server_key_version: "encryptionkey@0"
I20260929 16:34:03.784821 26641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "c0fd7ec8a3e243888fdf2362e406e972"
format_stamp: "Formatted at 2026-09-29 16:34:03 on dist-test-slave-dssx"
server_key: "5f8ee1f4f13574eb06c6274d580def1b"
server_key_iv: "0eb6df6a09900263b6ea3d0e06432631"
server_key_version: "encryptionkey@0"
I20260929 16:34:03.790025 26641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 16:34:03.793659 26656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:03.795061 26641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:03.795254 26641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "c0fd7ec8a3e243888fdf2362e406e972"
format_stamp: "Formatted at 2026-09-29 16:34:03 on dist-test-slave-dssx"
server_key: "5f8ee1f4f13574eb06c6274d580def1b"
server_key_iv: "0eb6df6a09900263b6ea3d0e06432631"
server_key_version: "encryptionkey@0"
I20260929 16:34:03.795465 26641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:03.801518 26641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:03.806197 26641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:03.806524 26641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:03.825659 26641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41467
I20260929 16:34:03.825682 26707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41467 every 8 connection(s)
I20260929 16:34:03.827188 26641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:03.830413 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 26641
I20260929 16:34:03.830626 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:03.831040 26708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:03.831112 6050 external_mini_cluster.cc:1428] Setting key 75a4cbdedb1f5ec12cec0d677227c531
I20260929 16:34:03.839594 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:03.832840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58111 (local address 127.0.0.1:41467)
0929 16:34:03.833661 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:03.833676 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:03.833727 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:34:03.835799 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:03.835826 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:03.836265 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:03.836692 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:03.837700 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:03.837712 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:03.837729 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:03.837745 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:03.837812 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:03.837848 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:03.838075 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:03.838213 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 16:34:03.838401 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":315,"threads_started":1}
I20260929 16:34:03.843297 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:03.845140 26708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972: Bootstrap starting.
I20260929 16:34:03.848367 26708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:03.849361 26708 log.cc:824] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:03.850428 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:03.847724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32811 (local address 127.0.0.1:41467)
0929 16:34:03.847883 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:03.847891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:03.848326 (+ 435us) server_negotiation.cc:407] Connection header received
0929 16:34:03.848519 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:03.848525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:03.848577 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:03.848704 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:03.849693 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:03.849699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:03.849705 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:03.849723 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:03.849764 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:03.849783 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:03.849910 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:03.850030 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 16:34:03.850094 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:34:03.853961 26708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972: No bootstrap required, opened a new log
I20260929 16:34:03.864492 26708 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fd7ec8a3e243888fdf2362e406e972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } }
I20260929 16:34:03.865087 26708 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:03.865195 26708 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd7ec8a3e243888fdf2362e406e972, State: Initialized, Role: FOLLOWER
I20260929 16:34:03.865954 26708 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fd7ec8a3e243888fdf2362e406e972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } }
I20260929 16:34:03.866250 26708 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:03.866394 26708 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:03.866569 26708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:03.870080 26708 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fd7ec8a3e243888fdf2362e406e972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } }
I20260929 16:34:03.870721 26708 leader_election.cc:302] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0fd7ec8a3e243888fdf2362e406e972; no voters:
I20260929 16:34:03.871433 26708 leader_election.cc:288] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:03.871608 26718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:03.872327 26718 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [term 1 LEADER]: Becoming Leader. State: Replica: c0fd7ec8a3e243888fdf2362e406e972, State: Running, Role: LEADER
I20260929 16:34:03.873003 26718 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fd7ec8a3e243888fdf2362e406e972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } }
I20260929 16:34:03.873893 26708 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:03.877069 26720 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0fd7ec8a3e243888fdf2362e406e972. Latest consensus state: current_term: 1 leader_uuid: "c0fd7ec8a3e243888fdf2362e406e972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fd7ec8a3e243888fdf2362e406e972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } } }
I20260929 16:34:03.877563 26720 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:03.879307 26725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:03.880968 26719 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0fd7ec8a3e243888fdf2362e406e972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0fd7ec8a3e243888fdf2362e406e972" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41467 } } }
I20260929 16:34:03.881407 26719 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:03.887037 26725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:03.899261 26725 catalog_manager.cc:1347] Generated new cluster ID: a03ac11a57864e7ab058a30e1e996591
I20260929 16:34:03.899459 26725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:03.925432 26725 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:03.925931 26725 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:03.942143 26725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0fd7ec8a3e243888fdf2362e406e972: Generated new TSK 0
I20260929 16:34:03.943238 26725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:03.952847 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26641
W20260929 16:34:03.954212 26703 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58111
W20260929 16:34:03.955052 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26641: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:03.955199 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 26641
2026-09-29T16:34:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:04 dist-test-slave-dssx krb5kdc[26739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:04 dist-test-slave-dssx krb5kdc[26739](info): set up 2 sockets
Sep 29 16:34:04 dist-test-slave-dssx krb5kdc[26739](info): commencing operation
krb5kdc: starting...
W20260929 16:34:05.061970 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:05 dist-test-slave-dssx krb5kdc[26739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:05.331036 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:05.670367 26752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:05.670739 26752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:05.670907 26752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:05.678489 26752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:05.678659 26752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:05.678772 26752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:05.678874 26752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:05.678958 26752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:05.689178 26752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:05.690609 26752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:05.691975 26752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:05.704566 26758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:05.704519 26757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:05.704480 26760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:05.705175 26752 server_base.cc:1034] running on GCE node
I20260929 16:34:05.706209 26752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:05.707734 26752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:05.708943 26752 hybrid_clock.cc:648] HybridClock initialized: now 1790699645708880 us; error 66 us; skew 500 ppm
I20260929 16:34:05.709276 26752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:05 dist-test-slave-dssx krb5kdc[26739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:05.712628 26752 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:05.714344 26752 webserver.cc:460] Webserver started at http://127.0.0.1:39915/ using document root <none> and password file <none>
I20260929 16:34:05.715092 26752 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:05.715227 26752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:05.715545 26752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:05.718652 26752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "09f217ff55954695a4c8095547f68895"
format_stamp: "Formatted at 2026-09-29 16:34:05 on dist-test-slave-dssx"
server_key: "db463788fdcf6f81d193ed6e6e08d2b0"
server_key_iv: "ffb9671c7a04b72a9bbfc5b807b808a8"
server_key_version: "encryptionkey@0"
I20260929 16:34:05.719463 26752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "09f217ff55954695a4c8095547f68895"
format_stamp: "Formatted at 2026-09-29 16:34:05 on dist-test-slave-dssx"
server_key: "db463788fdcf6f81d193ed6e6e08d2b0"
server_key_iv: "ffb9671c7a04b72a9bbfc5b807b808a8"
server_key_version: "encryptionkey@0"
I20260929 16:34:05.727288 26752 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260929 16:34:05.731333 26767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:05.732803 26752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:34:05.733019 26752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "09f217ff55954695a4c8095547f68895"
format_stamp: "Formatted at 2026-09-29 16:34:05 on dist-test-slave-dssx"
server_key: "db463788fdcf6f81d193ed6e6e08d2b0"
server_key_iv: "ffb9671c7a04b72a9bbfc5b807b808a8"
server_key_version: "encryptionkey@0"
I20260929 16:34:05.733234 26752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:05.750998 26752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:05.755111 26752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:05.755412 26752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:05.774185 26752 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42353
I20260929 16:34:05.774271 26818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42353 every 8 connection(s)
I20260929 16:34:05.775693 26752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:05.777082 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 26752
I20260929 16:34:05.777299 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:05.777897 6050 external_mini_cluster.cc:1428] Setting key f16c1da2d7e545abfbb9c7444422f89a
I20260929 16:34:05.780452 26819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:34:05 dist-test-slave-dssx krb5kdc[26739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:34:05.794239 26819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895: Bootstrap starting.
I20260929 16:34:05.797499 26819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:05.798540 26819 log.cc:824] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:05.801474 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:05.779688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55417 (local address 127.0.0.1:42353)
0929 16:34:05.780462 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:05.780473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:05.780535 (+ 62us) server_negotiation.cc:407] Connection header received
0929 16:34:05.782212 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:05.782234 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:05.782650 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:05.782996 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:05.784768 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:05.786229 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:05.788086 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:05.788818 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:05.788962 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:05.793964 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:05.793973 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:05.793992 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:05.794035 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:05.797142 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:05.798077 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:05.798084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:05.798091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:05.798269 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:05.798682 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:05.798689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:05.798692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:05.799216 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:05.799588 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:05.800005 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 16:34:05.800437 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":365,"threads_started":1}
I20260929 16:34:05.801635 26819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895: No bootstrap required, opened a new log
I20260929 16:34:05.805289 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:05.809731 26819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f217ff55954695a4c8095547f68895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260929 16:34:05.810271 26819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:05.810395 26819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f217ff55954695a4c8095547f68895, State: Initialized, Role: FOLLOWER
I20260929 16:34:05.811110 26819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f217ff55954695a4c8095547f68895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260929 16:34:05.811424 26819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:05.811566 26819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:05.811776 26819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:05.814883 26819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f217ff55954695a4c8095547f68895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260929 16:34:05.815508 26819 leader_election.cc:302] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f217ff55954695a4c8095547f68895; no voters:
I20260929 16:34:05.816192 26819 leader_election.cc:288] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:05.816259 26829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:05.818185 26819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:05.821034 26829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [term 1 LEADER]: Becoming Leader. State: Replica: 09f217ff55954695a4c8095547f68895, State: Running, Role: LEADER
I20260929 16:34:05.821722 26829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f217ff55954695a4c8095547f68895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } }
I20260929 16:34:05.824608 26830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f217ff55954695a4c8095547f68895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f217ff55954695a4c8095547f68895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
I20260929 16:34:05.825001 26830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:05.828754 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:05.809715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56957 (local address 127.0.0.1:42353)
0929 16:34:05.809895 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:05.809901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:05.810340 (+ 439us) server_negotiation.cc:407] Connection header received
0929 16:34:05.810551 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:05.810557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:05.810624 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:05.810732 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:05.812612 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:05.813454 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:05.815258 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:05.815753 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:05.815850 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:05.819657 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:05.819670 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:05.819678 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:05.819730 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:05.823821 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:05.824821 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:05.824832 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:05.824838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:05.825071 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:05.825678 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:05.825686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:05.825692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:05.826151 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:05.826525 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:05.827956 (+ 1431us) server_negotiation.cc:299] Negotiation successful
0929 16:34:05.828264 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:34:05.832770 26838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:05.832695 26831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f217ff55954695a4c8095547f68895. Latest consensus state: current_term: 1 leader_uuid: "09f217ff55954695a4c8095547f68895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f217ff55954695a4c8095547f68895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42353 } } }
I20260929 16:34:05.833050 26831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:05.837643 26838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:05.847512 26838 catalog_manager.cc:1347] Generated new cluster ID: e9cef321ea8648e88ffb05cc6afa228f
I20260929 16:34:05.847697 26838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:05.883685 26838 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:05.885593 26838 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:05.899143 26838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f217ff55954695a4c8095547f68895: Generated new TSK 0
I20260929 16:34:05.900261 26838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:05.911118 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26752
I20260929 16:34:05.912573 26814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:06.007643 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 26752
2026-09-29T16:34:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2052 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:06 dist-test-slave-dssx krb5kdc[26852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:06 dist-test-slave-dssx krb5kdc[26852](info): set up 2 sockets
Sep 29 16:34:06 dist-test-slave-dssx krb5kdc[26852](info): commencing operation
krb5kdc: starting...
W20260929 16:34:07.113983 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:07 dist-test-slave-dssx krb5kdc[26852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:07.374660 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:07.716113 26865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:07.716480 26865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:07.716670 26865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:07.726897 26865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:07.727064 26865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:07.727144 26865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:07.727242 26865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:07.727314 26865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:07.737946 26865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:07.739167 26865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:07.740408 26865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:07.752144 26871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:07.752182 26870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:07.752094 26873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:07.753145 26865 server_base.cc:1034] running on GCE node
I20260929 16:34:07.753980 26865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:07.755378 26865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:07.756578 26865 hybrid_clock.cc:648] HybridClock initialized: now 1790699647756522 us; error 54 us; skew 500 ppm
I20260929 16:34:07.756922 26865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:07 dist-test-slave-dssx krb5kdc[26852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:07.764914 26865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:07.766503 26865 webserver.cc:460] Webserver started at http://127.0.0.1:36553/ using document root <none> and password file <none>
I20260929 16:34:07.767225 26865 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:07.767359 26865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:07.767671 26865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:07.770726 26865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "f18401b07331483082b08d1b0f0a6979"
format_stamp: "Formatted at 2026-09-29 16:34:07 on dist-test-slave-dssx"
server_key: "12c47ccb63dae2c0589e461fe7b1c3b2"
server_key_iv: "7e1080d9d3ab1439c0ccf112c8b8da0f"
server_key_version: "encryptionkey@0"
I20260929 16:34:07.771555 26865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "f18401b07331483082b08d1b0f0a6979"
format_stamp: "Formatted at 2026-09-29 16:34:07 on dist-test-slave-dssx"
server_key: "12c47ccb63dae2c0589e461fe7b1c3b2"
server_key_iv: "7e1080d9d3ab1439c0ccf112c8b8da0f"
server_key_version: "encryptionkey@0"
I20260929 16:34:07.776806 26865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:34:07.780501 26880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:07.781925 26865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:07.782111 26865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "f18401b07331483082b08d1b0f0a6979"
format_stamp: "Formatted at 2026-09-29 16:34:07 on dist-test-slave-dssx"
server_key: "12c47ccb63dae2c0589e461fe7b1c3b2"
server_key_iv: "7e1080d9d3ab1439c0ccf112c8b8da0f"
server_key_version: "encryptionkey@0"
I20260929 16:34:07.782308 26865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:07.798426 26865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:07.802878 26865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:07.803167 26865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:07.822624 26865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38245
I20260929 16:34:07.822626 26931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38245 every 8 connection(s)
I20260929 16:34:07.824177 26865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:07.828099 26932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:07.829597 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 26865
I20260929 16:34:07.829808 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:07.830258 6050 external_mini_cluster.cc:1428] Setting key 38ee56e149f0c8ea72b46c35cd9be998
I20260929 16:34:07.841779 26932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979: Bootstrap starting.
I20260929 16:34:07.844098 26935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:07.832191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42097 (local address 127.0.0.1:38245)
0929 16:34:07.832980 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:07.832993 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:07.833034 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:34:07.834828 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:07.834855 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:07.835313 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:07.835743 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:07.836400 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:07.837745 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:07.839074 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:07.839976 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:07.840207 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:07.841494 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:07.841502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:07.841516 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:07.842139 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:07.842220 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:07.842246 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:07.842382 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:07.842613 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:34:07.842902 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":329,"threads_started":1}
I20260929 16:34:07.845742 26932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:07.846997 26932 log.cc:824] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:07.848420 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:07.850529 26932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979: No bootstrap required, opened a new log
I20260929 16:34:07.860776 26935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:07.854227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36803 (local address 127.0.0.1:38245)
0929 16:34:07.854433 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:07.854439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:07.854740 (+ 301us) server_negotiation.cc:407] Connection header received
0929 16:34:07.854903 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:07.854909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:07.854989 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:07.855110 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:07.855583 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:07.856686 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:07.858108 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:07.858855 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:07.858978 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:07.860016 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:07.860022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:07.860027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:07.860050 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:07.860101 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:07.860121 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:07.860216 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:07.860343 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 16:34:07.860440 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:34:07.861570 26932 raft_consensus.cc:348] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18401b07331483082b08d1b0f0a6979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38245 } }
I20260929 16:34:07.862165 26932 raft_consensus.cc:374] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:07.862308 26932 raft_consensus.cc:729] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f18401b07331483082b08d1b0f0a6979, State: Initialized, Role: FOLLOWER
I20260929 16:34:07.863124 26932 consensus_queue.cc:260] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18401b07331483082b08d1b0f0a6979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38245 } }
I20260929 16:34:07.863389 26932 raft_consensus.cc:388] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:07.863561 26932 raft_consensus.cc:482] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:07.863741 26932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:07.866046 26932 raft_consensus.cc:504] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18401b07331483082b08d1b0f0a6979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38245 } }
I20260929 16:34:07.866677 26932 leader_election.cc:302] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f18401b07331483082b08d1b0f0a6979; no voters:
I20260929 16:34:07.867252 26932 leader_election.cc:288] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:07.867381 26942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:07.868093 26942 raft_consensus.cc:686] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [term 1 LEADER]: Becoming Leader. State: Replica: f18401b07331483082b08d1b0f0a6979, State: Running, Role: LEADER
I20260929 16:34:07.868646 26942 consensus_queue.cc:237] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18401b07331483082b08d1b0f0a6979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38245 } }
I20260929 16:34:07.869482 26932 sys_catalog.cc:564] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:07.871551 26944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f18401b07331483082b08d1b0f0a6979. Latest consensus state: current_term: 1 leader_uuid: "f18401b07331483082b08d1b0f0a6979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18401b07331483082b08d1b0f0a6979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38245 } } }
I20260929 16:34:07.871567 26943 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f18401b07331483082b08d1b0f0a6979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18401b07331483082b08d1b0f0a6979" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38245 } } }
I20260929 16:34:07.871974 26944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:07.871992 26943 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:07.877241 26947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:07.881671 26947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:07.889755 26947 catalog_manager.cc:1347] Generated new cluster ID: f4de1f25fa154a62b51c37aa3fe26489
I20260929 16:34:07.889870 26947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:07.911526 26947 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:07.912891 26947 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:07.920389 26947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f18401b07331483082b08d1b0f0a6979: Generated new TSK 0
I20260929 16:34:07.921531 26947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:07.937013 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26865
W20260929 16:34:07.938145 26927 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42097
W20260929 16:34:07.938815 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26865: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:07.938961 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 26865
2026-09-29T16:34:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:08 dist-test-slave-dssx krb5kdc[26963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:08 dist-test-slave-dssx krb5kdc[26963](info): set up 2 sockets
Sep 29 16:34:08 dist-test-slave-dssx krb5kdc[26963](info): commencing operation
krb5kdc: starting...
W20260929 16:34:09.033932 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:09 dist-test-slave-dssx krb5kdc[26963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:09.294184 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:09.639810 26976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:09.640130 26976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:09.640300 26976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:09.647730 26976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:09.647866 26976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:09.647964 26976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:09.648072 26976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:09.648144 26976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:09.658244 26976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:09.660035 26976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:09.661317 26976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:09.673439 26981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:09.673462 26982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:09.673540 26984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:09.674062 26976 server_base.cc:1034] running on GCE node
I20260929 16:34:09.674950 26976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:09.676370 26976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:09.677569 26976 hybrid_clock.cc:648] HybridClock initialized: now 1790699649677519 us; error 32 us; skew 500 ppm
I20260929 16:34:09.677949 26976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:09 dist-test-slave-dssx krb5kdc[26963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:09.681253 26976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:09.682816 26976 webserver.cc:460] Webserver started at http://127.0.0.1:38643/ using document root <none> and password file <none>
I20260929 16:34:09.683497 26976 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:09.683601 26976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:09.683852 26976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:09.686966 26976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "9213733c42f54beb8ba774874d6cdd1c"
format_stamp: "Formatted at 2026-09-29 16:34:09 on dist-test-slave-dssx"
server_key: "91eedc8dfdab1bb1105b17d57dcff2ca"
server_key_iv: "18bdb0756728b2f695b0c27c2c325d92"
server_key_version: "encryptionkey@0"
I20260929 16:34:09.687721 26976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "9213733c42f54beb8ba774874d6cdd1c"
format_stamp: "Formatted at 2026-09-29 16:34:09 on dist-test-slave-dssx"
server_key: "91eedc8dfdab1bb1105b17d57dcff2ca"
server_key_iv: "18bdb0756728b2f695b0c27c2c325d92"
server_key_version: "encryptionkey@0"
I20260929 16:34:09.692937 26976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:34:09.696455 26991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:09.697901 26976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:09.698094 26976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "9213733c42f54beb8ba774874d6cdd1c"
format_stamp: "Formatted at 2026-09-29 16:34:09 on dist-test-slave-dssx"
server_key: "91eedc8dfdab1bb1105b17d57dcff2ca"
server_key_iv: "18bdb0756728b2f695b0c27c2c325d92"
server_key_version: "encryptionkey@0"
I20260929 16:34:09.698299 26976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:09.704511 26976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:09.709230 26976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:09.709633 26976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:09.735633 26976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38885
I20260929 16:34:09.735656 27042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38885 every 8 connection(s)
I20260929 16:34:09.737119 26976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:09.740105 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 26976
I20260929 16:34:09.740317 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:09.740773 6050 external_mini_cluster.cc:1428] Setting key bbc4f6a7d781319b3a713dff57e5d8e0
I20260929 16:34:09.741268 27043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:09.750437 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:09.742634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54709 (local address 127.0.0.1:38885)
0929 16:34:09.743482 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:09.743498 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:09.743551 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:34:09.746237 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:09.746269 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:09.746768 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:09.747297 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:09.748345 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:09.748354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:09.748370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:09.748389 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:09.748459 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:09.748500 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:09.748737 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:09.748904 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 16:34:09.749113 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":341,"threads_started":1}
I20260929 16:34:09.754215 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:09.755985 27043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c: Bootstrap starting.
I20260929 16:34:09.759883 27043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:09.761318 27043 log.cc:824] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:09.761439 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:09.758755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35257 (local address 127.0.0.1:38885)
0929 16:34:09.758982 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:09.758991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:09.759431 (+ 440us) server_negotiation.cc:407] Connection header received
0929 16:34:09.759629 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:09.759633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:09.759682 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:09.759767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:09.760730 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:09.760735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:09.760738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:09.760753 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:09.760792 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:09.760807 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:09.760896 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:09.761035 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 16:34:09.761091 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:34:09.764350 27043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c: No bootstrap required, opened a new log
I20260929 16:34:09.771813 27043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9213733c42f54beb8ba774874d6cdd1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260929 16:34:09.772238 27043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:09.772384 27043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9213733c42f54beb8ba774874d6cdd1c, State: Initialized, Role: FOLLOWER
I20260929 16:34:09.772977 27043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9213733c42f54beb8ba774874d6cdd1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260929 16:34:09.773180 27043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:09.773283 27043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:09.773502 27043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:09.775944 27043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9213733c42f54beb8ba774874d6cdd1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260929 16:34:09.776410 27043 leader_election.cc:302] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9213733c42f54beb8ba774874d6cdd1c; no voters:
I20260929 16:34:09.776948 27043 leader_election.cc:288] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:09.777104 27053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:09.777781 27053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [term 1 LEADER]: Becoming Leader. State: Replica: 9213733c42f54beb8ba774874d6cdd1c, State: Running, Role: LEADER
I20260929 16:34:09.778322 27053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9213733c42f54beb8ba774874d6cdd1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260929 16:34:09.779096 27043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:09.781236 27055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9213733c42f54beb8ba774874d6cdd1c. Latest consensus state: current_term: 1 leader_uuid: "9213733c42f54beb8ba774874d6cdd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9213733c42f54beb8ba774874d6cdd1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } } }
I20260929 16:34:09.781747 27055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:09.782572 27054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9213733c42f54beb8ba774874d6cdd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9213733c42f54beb8ba774874d6cdd1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } } }
I20260929 16:34:09.782981 27054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:09.784547 27061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:09.790184 27061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:09.799150 27061 catalog_manager.cc:1347] Generated new cluster ID: 1cd777ca73e54bcabbf7165d33b9b68e
I20260929 16:34:09.799275 27061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:09.841862 27061 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:09.842433 27061 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:09.861721 27061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9213733c42f54beb8ba774874d6cdd1c: Generated new TSK 0
I20260929 16:34:09.862619 27061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:09.882270 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26976
W20260929 16:34:09.884195 27038 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54709
W20260929 16:34:09.884944 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 26976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:09.885097 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 26976
2026-09-29T16:34:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:09 dist-test-slave-dssx krb5kdc[27074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:09 dist-test-slave-dssx krb5kdc[27074](info): set up 2 sockets
Sep 29 16:34:09 dist-test-slave-dssx krb5kdc[27074](info): commencing operation
krb5kdc: starting...
W20260929 16:34:10.985857 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:11 dist-test-slave-dssx krb5kdc[27074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:11.240517 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:11.592036 27087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:11.592351 27087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:11.592555 27087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:11.599844 27087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:11.599977 27087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:11.600108 27087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:11.600225 27087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:11.600313 27087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:11.610294 27087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:11.611692 27087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:11.613035 27087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:11.625315 27092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:11.625319 27093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:11.625370 27095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:11.626241 27087 server_base.cc:1034] running on GCE node
I20260929 16:34:11.627039 27087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:11.628495 27087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:11.629739 27087 hybrid_clock.cc:648] HybridClock initialized: now 1790699651629701 us; error 56 us; skew 500 ppm
I20260929 16:34:11.630115 27087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:11 dist-test-slave-dssx krb5kdc[27074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:11.633212 27087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:11.634884 27087 webserver.cc:460] Webserver started at http://127.0.0.1:32779/ using document root <none> and password file <none>
I20260929 16:34:11.635623 27087 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:11.635772 27087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:11.636077 27087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:11.639174 27087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "ea72c08e16374fab93cc1018d91998a5"
format_stamp: "Formatted at 2026-09-29 16:34:11 on dist-test-slave-dssx"
server_key: "9597c7806c468f4a06c54fa22e247bd7"
server_key_iv: "eb3f56022acf419aedacfd8497295034"
server_key_version: "encryptionkey@0"
I20260929 16:34:11.639963 27087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "ea72c08e16374fab93cc1018d91998a5"
format_stamp: "Formatted at 2026-09-29 16:34:11 on dist-test-slave-dssx"
server_key: "9597c7806c468f4a06c54fa22e247bd7"
server_key_iv: "eb3f56022acf419aedacfd8497295034"
server_key_version: "encryptionkey@0"
I20260929 16:34:11.645545 27087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:34:11.649214 27102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:11.650743 27087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:11.650960 27087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "ea72c08e16374fab93cc1018d91998a5"
format_stamp: "Formatted at 2026-09-29 16:34:11 on dist-test-slave-dssx"
server_key: "9597c7806c468f4a06c54fa22e247bd7"
server_key_iv: "eb3f56022acf419aedacfd8497295034"
server_key_version: "encryptionkey@0"
I20260929 16:34:11.651175 27087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:11.689256 27087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:11.693832 27087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:11.694161 27087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:11.713493 27087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46001
I20260929 16:34:11.713492 27153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46001 every 8 connection(s)
I20260929 16:34:11.715034 27087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:11.715725 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 27087
I20260929 16:34:11.715934 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:11.716526 6050 external_mini_cluster.cc:1428] Setting key bfbdedaa466ca5602cef6588040e51fd
I20260929 16:34:11.719636 27154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:34:11 dist-test-slave-dssx krb5kdc[27074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:34:11.733765 27154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5: Bootstrap starting.
I20260929 16:34:11.737282 27154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:11.738381 27154 log.cc:824] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:11.741469 27154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5: No bootstrap required, opened a new log
I20260929 16:34:11.742066 27157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:11.718362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:44071 (local address 127.0.0.1:46001)
0929 16:34:11.719195 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:11.719207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:11.719249 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:34:11.721932 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:11.721965 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:11.722645 (+ 680us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:11.723212 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:11.725481 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:11.727698 (+ 2217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:11.729038 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:11.729884 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:11.730083 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:11.734396 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:11.734404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:11.734420 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:11.734461 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:11.737592 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:11.738457 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:11.738464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:11.738470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:11.738684 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:11.739195 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:11.739200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:11.739203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:11.739753 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:11.740089 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:11.740621 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 16:34:11.740945 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":384,"threads_started":1}
I20260929 16:34:11.746423 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:11.750079 27154 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea72c08e16374fab93cc1018d91998a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46001 } }
I20260929 16:34:11.750515 27154 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:11.750636 27154 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea72c08e16374fab93cc1018d91998a5, State: Initialized, Role: FOLLOWER
I20260929 16:34:11.751425 27154 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea72c08e16374fab93cc1018d91998a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46001 } }
I20260929 16:34:11.751765 27154 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:11.751976 27154 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:11.752214 27154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:11.755445 27154 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea72c08e16374fab93cc1018d91998a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46001 } }
I20260929 16:34:11.756112 27154 leader_election.cc:302] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea72c08e16374fab93cc1018d91998a5; no voters:
I20260929 16:34:11.756815 27154 leader_election.cc:288] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:11.756930 27164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:11.757654 27164 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [term 1 LEADER]: Becoming Leader. State: Replica: ea72c08e16374fab93cc1018d91998a5, State: Running, Role: LEADER
I20260929 16:34:11.758378 27164 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea72c08e16374fab93cc1018d91998a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46001 } }
I20260929 16:34:11.760728 27154 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:11.761454 27166 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea72c08e16374fab93cc1018d91998a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea72c08e16374fab93cc1018d91998a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46001 } } }
I20260929 16:34:11.761888 27166 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:11.772241 27157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:11.751104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:38275 (local address 127.0.0.1:46001)
0929 16:34:11.751317 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:11.751327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:11.751816 (+ 489us) server_negotiation.cc:407] Connection header received
0929 16:34:11.752057 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:11.752061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:11.752156 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:11.752266 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:11.754327 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:11.755170 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:11.757074 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:11.757933 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:11.758062 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:11.761741 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:11.761753 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:11.761760 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:11.761801 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:11.766447 (+ 4646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:11.767841 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:11.767850 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:11.767857 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:11.768127 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:11.768898 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:11.768906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:11.768912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:11.769454 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:11.769923 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:11.771428 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0929 16:34:11.771704 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:34:11.762845 27165 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea72c08e16374fab93cc1018d91998a5. Latest consensus state: current_term: 1 leader_uuid: "ea72c08e16374fab93cc1018d91998a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea72c08e16374fab93cc1018d91998a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46001 } } }
I20260929 16:34:11.772588 27165 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:11.774624 27169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:11.780404 27169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:11.790571 27169 catalog_manager.cc:1347] Generated new cluster ID: bd2599220b674d178f8cab1bd068f025
I20260929 16:34:11.790719 27169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:11.801956 27169 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:11.803306 27169 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:11.822805 27169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea72c08e16374fab93cc1018d91998a5: Generated new TSK 0
I20260929 16:34:11.823714 27169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:11.830822 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27087
I20260929 16:34:11.832206 27149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:11.924940 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 27087
W20260929 16:34:11.943120 27072 connection.cc:535] client connection to 127.0.0.1:46001 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46001): Connection reset by peer (error 104)
2026-09-29T16:34:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:12 dist-test-slave-dssx krb5kdc[27187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:12 dist-test-slave-dssx krb5kdc[27187](info): set up 2 sockets
Sep 29 16:34:12 dist-test-slave-dssx krb5kdc[27187](info): commencing operation
krb5kdc: starting...
W20260929 16:34:13.026015 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:13 dist-test-slave-dssx krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:13.280462 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:13.618738 27200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:13.619026 27200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:13.619235 27200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:13.627130 27200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:13.627236 27200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:13.627355 27200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:13.627481 27200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:13.627542 27200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:13.637588 27200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:13.638840 27200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:13.640152 27200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:13.651845 27205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:13.651818 27208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:13.651966 27206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:13.652535 27200 server_base.cc:1034] running on GCE node
I20260929 16:34:13.653318 27200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:13.654664 27200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:13.655866 27200 hybrid_clock.cc:648] HybridClock initialized: now 1790699653655810 us; error 51 us; skew 500 ppm
I20260929 16:34:13.656213 27200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:13 dist-test-slave-dssx krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:13.666229 27200 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:13.667841 27200 webserver.cc:460] Webserver started at http://127.0.0.1:37671/ using document root <none> and password file <none>
I20260929 16:34:13.668531 27200 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:13.668670 27200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:13.668982 27200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:13.672004 27200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "30b77cb6befe4b0181ec2285d1aaced0"
format_stamp: "Formatted at 2026-09-29 16:34:13 on dist-test-slave-dssx"
server_key: "1a028f8b752daf0f4580bd90007fd095"
server_key_iv: "395fcb810a54c972c17a5deaf03cbca7"
server_key_version: "encryptionkey@0"
I20260929 16:34:13.672760 27200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "30b77cb6befe4b0181ec2285d1aaced0"
format_stamp: "Formatted at 2026-09-29 16:34:13 on dist-test-slave-dssx"
server_key: "1a028f8b752daf0f4580bd90007fd095"
server_key_iv: "395fcb810a54c972c17a5deaf03cbca7"
server_key_version: "encryptionkey@0"
I20260929 16:34:13.677881 27200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:34:13.681234 27215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:13.682600 27200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:13.682812 27200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "30b77cb6befe4b0181ec2285d1aaced0"
format_stamp: "Formatted at 2026-09-29 16:34:13 on dist-test-slave-dssx"
server_key: "1a028f8b752daf0f4580bd90007fd095"
server_key_iv: "395fcb810a54c972c17a5deaf03cbca7"
server_key_version: "encryptionkey@0"
I20260929 16:34:13.683012 27200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:13.703401 27200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:13.707845 27200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:13.708158 27200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:13.734215 27200 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34937
I20260929 16:34:13.734243 27266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34937 every 8 connection(s)
I20260929 16:34:13.735657 27200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:13.736749 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 27200
I20260929 16:34:13.736950 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:13.737437 6050 external_mini_cluster.cc:1428] Setting key 3028a5a15f0785256faa97ba2a55fabf
I20260929 16:34:13.740074 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:13.750975 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:13.739286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:50783 (local address 127.0.0.1:34937)
0929 16:34:13.740045 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:13.740056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:13.740098 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:34:13.741919 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:13.741942 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:13.742336 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:13.742680 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:13.743281 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:13.744580 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:13.746545 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:13.747196 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:13.747336 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:13.748611 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:13.748618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:13.748632 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:13.749186 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:13.749267 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:13.749291 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:13.749481 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:13.749769 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 16:34:13.749956 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":351,"threads_started":1}
I20260929 16:34:13.752236 27267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0: Bootstrap starting.
I20260929 16:34:13.754881 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:13.755609 27267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:13.756594 27267 log.cc:824] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:13.759889 27267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0: No bootstrap required, opened a new log
I20260929 16:34:13.765643 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:13.759331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:43371 (local address 127.0.0.1:34937)
0929 16:34:13.759520 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:13.759528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:13.759914 (+ 386us) server_negotiation.cc:407] Connection header received
0929 16:34:13.760170 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:13.760174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:13.760234 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:13.760359 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:13.760838 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:13.761717 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:13.763076 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:13.763551 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:13.763686 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:13.764869 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:13.764874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:13.764877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:13.764894 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:13.764933 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:13.764948 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:13.765075 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:13.765277 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 16:34:13.765339 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:34:13.767923 27267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b77cb6befe4b0181ec2285d1aaced0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34937 } }
I20260929 16:34:13.768325 27267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:13.768440 27267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b77cb6befe4b0181ec2285d1aaced0, State: Initialized, Role: FOLLOWER
I20260929 16:34:13.769012 27267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b77cb6befe4b0181ec2285d1aaced0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34937 } }
I20260929 16:34:13.769254 27267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:13.769460 27267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:13.769676 27267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:13.771935 27267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b77cb6befe4b0181ec2285d1aaced0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34937 } }
I20260929 16:34:13.772447 27267 leader_election.cc:302] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30b77cb6befe4b0181ec2285d1aaced0; no voters:
I20260929 16:34:13.773047 27267 leader_election.cc:288] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:13.773180 27277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:13.773943 27277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [term 1 LEADER]: Becoming Leader. State: Replica: 30b77cb6befe4b0181ec2285d1aaced0, State: Running, Role: LEADER
I20260929 16:34:13.774447 27277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b77cb6befe4b0181ec2285d1aaced0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34937 } }
I20260929 16:34:13.775296 27267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:13.777299 27279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30b77cb6befe4b0181ec2285d1aaced0. Latest consensus state: current_term: 1 leader_uuid: "30b77cb6befe4b0181ec2285d1aaced0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b77cb6befe4b0181ec2285d1aaced0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34937 } } }
I20260929 16:34:13.777761 27279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:13.777585 27278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30b77cb6befe4b0181ec2285d1aaced0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b77cb6befe4b0181ec2285d1aaced0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34937 } } }
I20260929 16:34:13.777853 27278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:13.778601 27283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:13.784403 27283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:13.795089 27283 catalog_manager.cc:1347] Generated new cluster ID: a432f11c0c7c4fa58696091284326f09
I20260929 16:34:13.795245 27283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:13.803045 27283 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:13.804347 27283 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:13.811232 27283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30b77cb6befe4b0181ec2285d1aaced0: Generated new TSK 0
I20260929 16:34:13.812147 27283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:13.831190 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27200
W20260929 16:34:13.832386 27262 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.38:50783
W20260929 16:34:13.833118 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27200: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:13.833278 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 27200
W20260929 16:34:13.850176 27185 connection.cc:535] client connection to 127.0.0.1:34937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34937): Connection reset by peer (error 104)
2026-09-29T16:34:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1906 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:13 dist-test-slave-dssx krb5kdc[27298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:13 dist-test-slave-dssx krb5kdc[27298](info): set up 2 sockets
Sep 29 16:34:13 dist-test-slave-dssx krb5kdc[27298](info): commencing operation
krb5kdc: starting...
W20260929 16:34:14.957875 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:15 dist-test-slave-dssx krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:15.226152 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:15.563655 27311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:15.563956 27311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:15.564049 27311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:15.571862 27311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:15.571990 27311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:15.572045 27311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:15.572105 27311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:15.572151 27311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:15.582382 27311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:15.583652 27311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:15.584942 27311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:15.596828 27317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:15.596843 27316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:15.596835 27319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:15.598577 27311 server_base.cc:1034] running on GCE node
I20260929 16:34:15.599393 27311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:15.601048 27311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:15.602280 27311 hybrid_clock.cc:648] HybridClock initialized: now 1790699655602219 us; error 58 us; skew 500 ppm
I20260929 16:34:15.602638 27311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:15 dist-test-slave-dssx krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:15.605831 27311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:15.607375 27311 webserver.cc:460] Webserver started at http://127.0.0.1:42533/ using document root <none> and password file <none>
I20260929 16:34:15.608095 27311 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:15.608255 27311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:15.608569 27311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:15.611789 27311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "7864ca0b97494d0488d95e8694be1145"
format_stamp: "Formatted at 2026-09-29 16:34:15 on dist-test-slave-dssx"
server_key: "f49a59c0f4a31176abf938a4418e2fb3"
server_key_iv: "55a8663a166de6ced527a1ca5cd1f26b"
server_key_version: "encryptionkey@0"
I20260929 16:34:15.612591 27311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "7864ca0b97494d0488d95e8694be1145"
format_stamp: "Formatted at 2026-09-29 16:34:15 on dist-test-slave-dssx"
server_key: "f49a59c0f4a31176abf938a4418e2fb3"
server_key_iv: "55a8663a166de6ced527a1ca5cd1f26b"
server_key_version: "encryptionkey@0"
I20260929 16:34:15.618515 27311 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 16:34:15.622368 27326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:15.623802 27311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:34:15.624008 27311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "7864ca0b97494d0488d95e8694be1145"
format_stamp: "Formatted at 2026-09-29 16:34:15 on dist-test-slave-dssx"
server_key: "f49a59c0f4a31176abf938a4418e2fb3"
server_key_iv: "55a8663a166de6ced527a1ca5cd1f26b"
server_key_version: "encryptionkey@0"
I20260929 16:34:15.624238 27311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:15.630407 27311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:15.635109 27311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:15.635443 27311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:15.654577 27311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39491
I20260929 16:34:15.654613 27377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39491 every 8 connection(s)
I20260929 16:34:15.656044 27311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:15.659924 27378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:15.661779 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 27311
I20260929 16:34:15.661965 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:15.662427 6050 external_mini_cluster.cc:1428] Setting key deb073eade893b5c81d3128e6ba40599
I20260929 16:34:15.670521 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:15.664309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:58453 (local address 127.0.0.1:39491)
0929 16:34:15.665078 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:15.665093 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:15.665140 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:34:15.666832 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:15.666856 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:15.667246 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:15.667656 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:15.668807 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:15.668814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:15.668827 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:15.668843 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:15.668912 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:15.668934 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:15.669170 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:15.669324 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 16:34:15.669548 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":333,"threads_started":1}
I20260929 16:34:15.673367 27378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145: Bootstrap starting.
I20260929 16:34:15.674328 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:15.676633 27378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:15.677881 27378 log.cc:824] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:15.681164 27378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145: No bootstrap required, opened a new log
I20260929 16:34:15.681478 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:15.678766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:45073 (local address 127.0.0.1:39491)
0929 16:34:15.678974 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:15.678982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:15.679469 (+ 487us) server_negotiation.cc:407] Connection header received
0929 16:34:15.679593 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:15.679598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:15.679654 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:15.679778 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:15.680728 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:15.680733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:15.680737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:15.680756 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:15.680805 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:15.680825 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:15.680948 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:15.681025 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 16:34:15.681086 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:34:15.689082 27378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7864ca0b97494d0488d95e8694be1145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260929 16:34:15.689539 27378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:15.689622 27378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7864ca0b97494d0488d95e8694be1145, State: Initialized, Role: FOLLOWER
I20260929 16:34:15.690253 27378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7864ca0b97494d0488d95e8694be1145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260929 16:34:15.690464 27378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:15.690565 27378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:15.690692 27378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:15.692836 27378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7864ca0b97494d0488d95e8694be1145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260929 16:34:15.693555 27378 leader_election.cc:302] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7864ca0b97494d0488d95e8694be1145; no voters:
I20260929 16:34:15.694106 27378 leader_election.cc:288] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:15.694270 27388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:15.694979 27388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [term 1 LEADER]: Becoming Leader. State: Replica: 7864ca0b97494d0488d95e8694be1145, State: Running, Role: LEADER
I20260929 16:34:15.695526 27388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7864ca0b97494d0488d95e8694be1145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260929 16:34:15.696244 27378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:15.698628 27390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7864ca0b97494d0488d95e8694be1145. Latest consensus state: current_term: 1 leader_uuid: "7864ca0b97494d0488d95e8694be1145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7864ca0b97494d0488d95e8694be1145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } } }
I20260929 16:34:15.698752 27389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7864ca0b97494d0488d95e8694be1145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7864ca0b97494d0488d95e8694be1145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } } }
I20260929 16:34:15.699066 27390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:15.699158 27389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:15.701035 27393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:15.707979 27393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:15.717435 27393 catalog_manager.cc:1347] Generated new cluster ID: d64e1d3950e2456896c58ec0e3d69c7d
I20260929 16:34:15.717566 27393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:15.732558 27393 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:15.733098 27393 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:15.780977 27393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7864ca0b97494d0488d95e8694be1145: Generated new TSK 0
I20260929 16:34:15.782217 27393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:15.793008 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27311
W20260929 16:34:15.794488 27373 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.38:58453
W20260929 16:34:15.795316 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27311: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:15.795490 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 27311
2026-09-29T16:34:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:15 dist-test-slave-dssx krb5kdc[27409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:15 dist-test-slave-dssx krb5kdc[27409](info): set up 2 sockets
Sep 29 16:34:15 dist-test-slave-dssx krb5kdc[27409](info): commencing operation
krb5kdc: starting...
W20260929 16:34:16.902025 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:17 dist-test-slave-dssx krb5kdc[27409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:17.168341 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:17.504108 27422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:17.504417 27422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:17.504580 27422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:17.512096 27422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:17.512223 27422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:17.512346 27422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:17.512454 27422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:17.512537 27422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:17.522832 27422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:17.524231 27422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:17.525588 27422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:17.537654 27428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:17.537736 27430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:17.537704 27427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:17.538547 27422 server_base.cc:1034] running on GCE node
I20260929 16:34:17.539299 27422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:17.540733 27422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:17.541919 27422 hybrid_clock.cc:648] HybridClock initialized: now 1790699657541856 us; error 53 us; skew 500 ppm
I20260929 16:34:17.542284 27422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:17 dist-test-slave-dssx krb5kdc[27409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:17.545493 27422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:17.547013 27422 webserver.cc:460] Webserver started at http://127.0.0.1:42981/ using document root <none> and password file <none>
I20260929 16:34:17.547752 27422 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:17.547905 27422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:17.548224 27422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:17.551376 27422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "5a2dc85388c24a24b9f93df0efad2200"
format_stamp: "Formatted at 2026-09-29 16:34:17 on dist-test-slave-dssx"
server_key: "223b23491ec5c09d72f094aa1d58acf2"
server_key_iv: "f33aeeb9ef155cd411d7200c95089a08"
server_key_version: "encryptionkey@0"
I20260929 16:34:17.552208 27422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "5a2dc85388c24a24b9f93df0efad2200"
format_stamp: "Formatted at 2026-09-29 16:34:17 on dist-test-slave-dssx"
server_key: "223b23491ec5c09d72f094aa1d58acf2"
server_key_iv: "f33aeeb9ef155cd411d7200c95089a08"
server_key_version: "encryptionkey@0"
I20260929 16:34:17.557703 27422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:34:17.561286 27437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:17.562841 27422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:17.563058 27422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "5a2dc85388c24a24b9f93df0efad2200"
format_stamp: "Formatted at 2026-09-29 16:34:17 on dist-test-slave-dssx"
server_key: "223b23491ec5c09d72f094aa1d58acf2"
server_key_iv: "f33aeeb9ef155cd411d7200c95089a08"
server_key_version: "encryptionkey@0"
I20260929 16:34:17.563285 27422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:17.583256 27422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:17.587567 27422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:17.587886 27422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:17.606992 27422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36451
I20260929 16:34:17.607021 27488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36451 every 8 connection(s)
I20260929 16:34:17.608461 27422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:17.612315 27489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:17.613360 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 27422
I20260929 16:34:17.613572 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:17.614012 6050 external_mini_cluster.cc:1428] Setting key 0811096334efeab758dabe80377286d8
I20260929 16:34:17.626060 27489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200: Bootstrap starting.
Sep 29 16:34:17 dist-test-slave-dssx krb5kdc[27409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:34:17.629451 27489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:17.630751 27489 log.cc:824] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:17.633786 27489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200: No bootstrap required, opened a new log
I20260929 16:34:17.637872 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:17.615964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:49133 (local address 127.0.0.1:36451)
0929 16:34:17.616785 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:17.616806 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:17.616859 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:34:17.619069 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:17.619100 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:17.619686 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:17.620168 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:17.621557 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:17.622881 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:17.624197 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:17.624899 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:17.625110 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:17.629612 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:17.629624 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:17.629650 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:17.629706 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:17.632968 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:17.633919 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:17.634087 (+ 168us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:17.634098 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:17.634324 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:17.634899 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:17.634903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:17.634905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:17.635465 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:17.635844 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:17.636399 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 16:34:17.636742 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":342,"threads_started":1}
I20260929 16:34:17.641983 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:17.642522 27489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2dc85388c24a24b9f93df0efad2200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36451 } }
I20260929 16:34:17.642922 27489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:17.643035 27489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2dc85388c24a24b9f93df0efad2200, State: Initialized, Role: FOLLOWER
I20260929 16:34:17.643676 27489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2dc85388c24a24b9f93df0efad2200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36451 } }
I20260929 16:34:17.643929 27489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:17.644079 27489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:17.644269 27489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:17.646708 27489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2dc85388c24a24b9f93df0efad2200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36451 } }
I20260929 16:34:17.647471 27489 leader_election.cc:302] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a2dc85388c24a24b9f93df0efad2200; no voters:
I20260929 16:34:17.648159 27489 leader_election.cc:288] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:17.648227 27499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:17.650056 27499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [term 1 LEADER]: Becoming Leader. State: Replica: 5a2dc85388c24a24b9f93df0efad2200, State: Running, Role: LEADER
I20260929 16:34:17.650695 27489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:17.650758 27499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2dc85388c24a24b9f93df0efad2200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36451 } }
I20260929 16:34:17.653672 27500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a2dc85388c24a24b9f93df0efad2200" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2dc85388c24a24b9f93df0efad2200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36451 } } }
I20260929 16:34:17.654078 27500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:17.658277 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a2dc85388c24a24b9f93df0efad2200. Latest consensus state: current_term: 1 leader_uuid: "5a2dc85388c24a24b9f93df0efad2200" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a2dc85388c24a24b9f93df0efad2200" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36451 } } }
I20260929 16:34:17.658842 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:17.662200 27505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:17.667658 27505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:17.669972 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:17.646621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:56921 (local address 127.0.0.1:36451)
0929 16:34:17.646880 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:17.646890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:17.647292 (+ 402us) server_negotiation.cc:407] Connection header received
0929 16:34:17.647475 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:17.647478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:17.647544 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:17.647664 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:17.649789 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:17.651108 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:17.653273 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:17.654324 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:17.654490 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:17.658910 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:17.658921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:17.658928 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:17.658987 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:17.664085 (+ 5098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:17.665561 (+ 1476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:17.665570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:17.665580 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:17.665854 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:17.666615 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:17.666623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:17.666629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:17.667286 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:17.668226 (+ 940us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:17.669128 (+ 902us) server_negotiation.cc:299] Negotiation successful
0929 16:34:17.669364 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:34:17.678791 27505 catalog_manager.cc:1347] Generated new cluster ID: 306bab4de4b448c186292c2c02d8d0f1
I20260929 16:34:17.678928 27505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:17.723532 27505 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:17.725488 27505 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:17.737468 27505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a2dc85388c24a24b9f93df0efad2200: Generated new TSK 0
I20260929 16:34:17.738426 27505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:17.749147 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27422
I20260929 16:34:17.750419 27484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:17.845573 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 27422
W20260929 16:34:17.862905 27407 connection.cc:535] client connection to 127.0.0.1:36451 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36451): Connection reset by peer (error 104)
2026-09-29T16:34:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:17 dist-test-slave-dssx krb5kdc[27521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:17 dist-test-slave-dssx krb5kdc[27521](info): set up 2 sockets
Sep 29 16:34:17 dist-test-slave-dssx krb5kdc[27521](info): commencing operation
krb5kdc: starting...
W20260929 16:34:18.957828 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:19 dist-test-slave-dssx krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:19.228353 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:19.580106 27534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:19.580498 27534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:19.580677 27534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:19.588269 27534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:19.588459 27534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:19.588584 27534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:19.588696 27534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:19.588783 27534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:19.599131 27534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:19.600571 27534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:19.602039 27534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:19.614511 27539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:19.614436 27540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:19.614459 27542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:19.615720 27534 server_base.cc:1034] running on GCE node
I20260929 16:34:19.616601 27534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:19.618175 27534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:19.619402 27534 hybrid_clock.cc:648] HybridClock initialized: now 1790699659619333 us; error 59 us; skew 500 ppm
I20260929 16:34:19.619814 27534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:19 dist-test-slave-dssx krb5kdc[27521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:19.626348 27534 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:19.627935 27534 webserver.cc:460] Webserver started at http://127.0.0.1:40707/ using document root <none> and password file <none>
I20260929 16:34:19.628639 27534 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:19.628765 27534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:19.629030 27534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:19.632269 27534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "3d2e3d609fa143e6b16c8e45c2544e98"
format_stamp: "Formatted at 2026-09-29 16:34:19 on dist-test-slave-dssx"
server_key: "bdfb364546cd961316a198b68a26d118"
server_key_iv: "ecba78d9bcc9b6d964b473fd9594dbbf"
server_key_version: "encryptionkey@0"
I20260929 16:34:19.633046 27534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "3d2e3d609fa143e6b16c8e45c2544e98"
format_stamp: "Formatted at 2026-09-29 16:34:19 on dist-test-slave-dssx"
server_key: "bdfb364546cd961316a198b68a26d118"
server_key_iv: "ecba78d9bcc9b6d964b473fd9594dbbf"
server_key_version: "encryptionkey@0"
I20260929 16:34:19.638960 27534 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:34:19.642645 27549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:19.644117 27534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:34:19.644330 27534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "3d2e3d609fa143e6b16c8e45c2544e98"
format_stamp: "Formatted at 2026-09-29 16:34:19 on dist-test-slave-dssx"
server_key: "bdfb364546cd961316a198b68a26d118"
server_key_iv: "ecba78d9bcc9b6d964b473fd9594dbbf"
server_key_version: "encryptionkey@0"
I20260929 16:34:19.644544 27534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:19.663184 27534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:19.668032 27534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:19.668365 27534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:19.688205 27534 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45131
I20260929 16:34:19.688241 27600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45131 every 8 connection(s)
I20260929 16:34:19.689845 27534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:19.693856 27601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:19.696008 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 27534
I20260929 16:34:19.696213 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:19.696744 6050 external_mini_cluster.cc:1428] Setting key 97d11c6f6ce7bc393c8bb29ca00cfb32
I20260929 16:34:19.708274 27601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98: Bootstrap starting.
I20260929 16:34:19.712476 27601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:19.713927 27601 log.cc:824] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:19.714679 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:19.699412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:49659 (local address 127.0.0.1:45131)
0929 16:34:19.700466 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:19.700484 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:19.700917 (+ 433us) server_negotiation.cc:407] Connection header received
0929 16:34:19.703498 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:19.703535 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:19.704191 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:19.704705 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:19.705299 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:19.707354 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:19.708909 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:19.709996 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:19.710234 (+ 238us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:19.711499 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:19.711513 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:19.711534 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:19.712278 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:19.712392 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:19.712429 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:19.712597 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:19.713011 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 16:34:19.713203 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":446,"threads_started":1}
I20260929 16:34:19.717065 27601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98: No bootstrap required, opened a new log
I20260929 16:34:19.718832 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:19.729490 27601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e3d609fa143e6b16c8e45c2544e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260929 16:34:19.730105 27601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:19.730234 27601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2e3d609fa143e6b16c8e45c2544e98, State: Initialized, Role: FOLLOWER
I20260929 16:34:19.731026 27601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e3d609fa143e6b16c8e45c2544e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260929 16:34:19.731321 27601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:19.731459 27601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:19.731647 27601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:19.734948 27601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e3d609fa143e6b16c8e45c2544e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260929 16:34:19.735659 27601 leader_election.cc:302] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d2e3d609fa143e6b16c8e45c2544e98; no voters:
I20260929 16:34:19.736505 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:19.727946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:58667 (local address 127.0.0.1:45131)
0929 16:34:19.728228 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:19.728235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:19.728991 (+ 756us) server_negotiation.cc:407] Connection header received
0929 16:34:19.729315 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:19.729321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:19.729458 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:19.729582 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:19.730350 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:19.731541 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:19.733675 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:19.734161 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:19.734276 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:19.735547 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:19.735552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:19.735556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:19.735576 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:19.735618 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:19.735634 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:19.735730 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:19.736053 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 16:34:19.736104 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260929 16:34:19.736706 27601 leader_election.cc:288] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:19.736822 27611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:19.737555 27611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [term 1 LEADER]: Becoming Leader. State: Replica: 3d2e3d609fa143e6b16c8e45c2544e98, State: Running, Role: LEADER
I20260929 16:34:19.738186 27611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e3d609fa143e6b16c8e45c2544e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260929 16:34:19.740306 27601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:19.741225 27612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d2e3d609fa143e6b16c8e45c2544e98. Latest consensus state: current_term: 1 leader_uuid: "3d2e3d609fa143e6b16c8e45c2544e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e3d609fa143e6b16c8e45c2544e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } } }
I20260929 16:34:19.741703 27612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:19.741575 27613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d2e3d609fa143e6b16c8e45c2544e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2e3d609fa143e6b16c8e45c2544e98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } } }
I20260929 16:34:19.741945 27613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:19.744129 27616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:19.749598 27616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:19.761570 27616 catalog_manager.cc:1347] Generated new cluster ID: bbf0ed31f8c8478ba80f48d47b77643d
I20260929 16:34:19.761706 27616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:19.770823 27616 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:19.772833 27616 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:19.781796 27616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d2e3d609fa143e6b16c8e45c2544e98: Generated new TSK 0
I20260929 16:34:19.782747 27616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:19.795032 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27534
W20260929 16:34:19.796903 27596 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.38:49659
W20260929 16:34:19.797885 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27534: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:19.798113 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 27534
W20260929 16:34:19.817864 27519 connection.cc:535] client connection to 127.0.0.1:45131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45131): Connection reset by peer (error 104)
2026-09-29T16:34:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:19 dist-test-slave-dssx krb5kdc[27632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:19 dist-test-slave-dssx krb5kdc[27632](info): set up 2 sockets
Sep 29 16:34:19 dist-test-slave-dssx krb5kdc[27632](info): commencing operation
krb5kdc: starting...
W20260929 16:34:20.921975 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:21 dist-test-slave-dssx krb5kdc[27632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:34:21.189198 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:21.535483 27645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:21.535878 27645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:21.536055 27645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:21.544251 27645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:21.544382 27645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:21.544608 27645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:21.544729 27645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:21.544819 27645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:21.555176 27645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:21.556614 27645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:21.558048 27645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:21.570906 27650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:21.570887 27653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:21.570834 27651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:21.571790 27645 server_base.cc:1034] running on GCE node
I20260929 16:34:21.572635 27645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:21.574255 27645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:21.575546 27645 hybrid_clock.cc:648] HybridClock initialized: now 1790699661575503 us; error 58 us; skew 500 ppm
I20260929 16:34:21.575974 27645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:21 dist-test-slave-dssx krb5kdc[27632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:21.579283 27645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:34:21.580946 27645 webserver.cc:460] Webserver started at http://127.0.0.1:39045/ using document root <none> and password file <none>
I20260929 16:34:21.581737 27645 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:21.581888 27645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:21.582204 27645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:21.585355 27645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "97934799f2a443059267668c1942c631"
format_stamp: "Formatted at 2026-09-29 16:34:21 on dist-test-slave-dssx"
server_key: "73f85f116748fbff8629884b51692a84"
server_key_iv: "874067362f16a1d96ee1bf3ef61324aa"
server_key_version: "encryptionkey@0"
I20260929 16:34:21.586268 27645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "97934799f2a443059267668c1942c631"
format_stamp: "Formatted at 2026-09-29 16:34:21 on dist-test-slave-dssx"
server_key: "73f85f116748fbff8629884b51692a84"
server_key_iv: "874067362f16a1d96ee1bf3ef61324aa"
server_key_version: "encryptionkey@0"
I20260929 16:34:21.591921 27645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:21.595955 27660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:21.597585 27645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:21.597796 27645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "97934799f2a443059267668c1942c631"
format_stamp: "Formatted at 2026-09-29 16:34:21 on dist-test-slave-dssx"
server_key: "73f85f116748fbff8629884b51692a84"
server_key_iv: "874067362f16a1d96ee1bf3ef61324aa"
server_key_version: "encryptionkey@0"
I20260929 16:34:21.598060 27645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:21.604821 27645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:21.610236 27645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:21.610513 27645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:21.630247 27645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36945
I20260929 16:34:21.630280 27711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36945 every 8 connection(s)
I20260929 16:34:21.631827 27645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:21.635912 27712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:21.635950 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 27645
I20260929 16:34:21.636174 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:21.636642 6050 external_mini_cluster.cc:1428] Setting key 59d2753b4d62d1d5ac03a2617b4300ae
I20260929 16:34:21.644851 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:21.638540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:52917 (local address 127.0.0.1:36945)
0929 16:34:21.639478 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:21.639493 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:21.639542 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:34:21.641122 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:21.641148 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:21.641634 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:21.642027 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:21.643154 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:21.643161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:21.643179 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:21.643198 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:21.643261 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:21.643281 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:21.643469 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:21.643591 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 16:34:21.643837 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":426,"threads_started":1}
I20260929 16:34:21.649142 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:21.650609 27712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631: Bootstrap starting.
I20260929 16:34:21.654557 27712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:21.655843 27712 log.cc:824] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:21.656477 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:21.653752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.38:47663 (local address 127.0.0.1:36945)
0929 16:34:21.653925 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:21.653930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:21.654395 (+ 465us) server_negotiation.cc:407] Connection header received
0929 16:34:21.654625 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:21.654637 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:21.654715 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:21.654820 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:21.655813 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:21.655817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:21.655820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:21.655838 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:21.655880 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:21.655896 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:21.656030 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:21.656148 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 16:34:21.656216 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:34:21.659155 27712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631: No bootstrap required, opened a new log
I20260929 16:34:21.667128 27712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97934799f2a443059267668c1942c631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260929 16:34:21.667580 27712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:21.667657 27712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97934799f2a443059267668c1942c631, State: Initialized, Role: FOLLOWER
I20260929 16:34:21.668381 27712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97934799f2a443059267668c1942c631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260929 16:34:21.668655 27712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:21.668834 27712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:21.669023 27712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:21.671417 27712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97934799f2a443059267668c1942c631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260929 16:34:21.671952 27712 leader_election.cc:302] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97934799f2a443059267668c1942c631; no voters:
I20260929 16:34:21.672538 27712 leader_election.cc:288] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:21.672705 27722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:21.673456 27722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [term 1 LEADER]: Becoming Leader. State: Replica: 97934799f2a443059267668c1942c631, State: Running, Role: LEADER
I20260929 16:34:21.674047 27722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97934799f2a443059267668c1942c631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260929 16:34:21.674832 27712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:21.677011 27724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97934799f2a443059267668c1942c631. Latest consensus state: current_term: 1 leader_uuid: "97934799f2a443059267668c1942c631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97934799f2a443059267668c1942c631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } } }
I20260929 16:34:21.677491 27724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:21.677868 27723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97934799f2a443059267668c1942c631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97934799f2a443059267668c1942c631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } } }
I20260929 16:34:21.678160 27723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:21.679823 27728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:21.685210 27728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:21.694568 27728 catalog_manager.cc:1347] Generated new cluster ID: 35afb9e943a9485184f7b5434d660c77
I20260929 16:34:21.694783 27728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:21.713646 27728 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:21.714169 27728 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:21.725936 27728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97934799f2a443059267668c1942c631: Generated new TSK 0
I20260929 16:34:21.727152 27728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:21.743430 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27645
W20260929 16:34:21.744918 27707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.38:52917
W20260929 16:34:21.745798 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27645: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:21.746104 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 27645
2026-09-29T16:34:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:21 dist-test-slave-dssx krb5kdc[27743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:21 dist-test-slave-dssx krb5kdc[27743](info): set up 2 sockets
Sep 29 16:34:21 dist-test-slave-dssx krb5kdc[27743](info): commencing operation
krb5kdc: starting...
W20260929 16:34:22.841970 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:23 dist-test-slave-dssx krb5kdc[27743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:34:23.112453 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:37409
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:23.461357 27756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:23.461840 27756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:23.462010 27756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:23.469662 27756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:23.469858 27756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:23.469969 27756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:23.470080 27756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:23.470158 27756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:23.480417 27756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:37409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:23.481905 27756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:23.483315 27756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:23.495783 27764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:23.495990 27762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:23.496039 27761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:23.497134 27756 server_base.cc:1034] running on GCE node
I20260929 16:34:23.498332 27756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:23.499930 27756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:23.501175 27756 hybrid_clock.cc:648] HybridClock initialized: now 1790699663501066 us; error 68 us; skew 500 ppm
I20260929 16:34:23.501731 27756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:23 dist-test-slave-dssx krb5kdc[27743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:23.505856 27756 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:34:23.507568 27756 webserver.cc:460] Webserver started at http://127.5.232.190:40737/ using document root <none> and password file <none>
I20260929 16:34:23.508436 27756 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:23.508610 27756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:23.508965 27756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:23.512357 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "0717054e3f0d4411b538aebeeeee5d1f"
format_stamp: "Formatted at 2026-09-29 16:34:23 on dist-test-slave-dssx"
server_key: "6c1890d7676ad6f80119c87649e86529"
server_key_iv: "01acea34409040a78609d657a0c76162"
server_key_version: "encryptionkey@0"
I20260929 16:34:23.513250 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "0717054e3f0d4411b538aebeeeee5d1f"
format_stamp: "Formatted at 2026-09-29 16:34:23 on dist-test-slave-dssx"
server_key: "6c1890d7676ad6f80119c87649e86529"
server_key_iv: "01acea34409040a78609d657a0c76162"
server_key_version: "encryptionkey@0"
I20260929 16:34:23.519122 27756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:34:23.523028 27771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:23.524498 27756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:23.524718 27756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "0717054e3f0d4411b538aebeeeee5d1f"
format_stamp: "Formatted at 2026-09-29 16:34:23 on dist-test-slave-dssx"
server_key: "6c1890d7676ad6f80119c87649e86529"
server_key_iv: "01acea34409040a78609d657a0c76162"
server_key_version: "encryptionkey@0"
I20260929 16:34:23.524959 27756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:23.543141 27756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:23.547742 27756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:23.548084 27756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:23.568202 27756 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:37409
I20260929 16:34:23.568208 27822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:37409 every 8 connection(s)
I20260929 16:34:23.569816 27756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:23.574263 27823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:23.579668 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 27756
I20260929 16:34:23.579911 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:23.580343 6050 external_mini_cluster.cc:1428] Setting key 4632bafd4d40fcd22b33e25c63c24f03
I20260929 16:34:23.586774 27823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f: Bootstrap starting.
I20260929 16:34:23.590335 27823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:23.591485 27823 log.cc:824] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:23.594669 27823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f: No bootstrap required, opened a new log
Sep 29 16:34:23 dist-test-slave-dssx krb5kdc[27743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:34:23.603057 27823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0717054e3f0d4411b538aebeeeee5d1f" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37409 } }
I20260929 16:34:23.603499 27823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:23.603641 27823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0717054e3f0d4411b538aebeeeee5d1f, State: Initialized, Role: FOLLOWER
I20260929 16:34:23.604362 27823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0717054e3f0d4411b538aebeeeee5d1f" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37409 } }
I20260929 16:34:23.604640 27823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:23.604799 27823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:23.605012 27823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:23.605767 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:23.582223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47451 (local address 127.5.232.190:37409)
0929 16:34:23.583222 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:23.583245 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:23.583583 (+ 338us) server_negotiation.cc:407] Connection header received
0929 16:34:23.585734 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:23.585762 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:23.586264 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:23.586711 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:23.588332 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:23.589822 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:23.591263 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:23.592088 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:23.592320 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:23.597012 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:23.597020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:23.597041 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:23.597086 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:23.600613 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:23.601649 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:23.601660 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:23.601669 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:23.601870 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:23.602441 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:23.602445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:23.602447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:23.603051 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:23.603429 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:23.603948 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 16:34:23.604365 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":426,"threads_started":1}
I20260929 16:34:23.608135 27823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0717054e3f0d4411b538aebeeeee5d1f" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37409 } }
I20260929 16:34:23.608771 27823 leader_election.cc:302] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0717054e3f0d4411b538aebeeeee5d1f; no voters:
I20260929 16:34:23.609455 27823 leader_election.cc:288] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:23.609601 27828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:23.610316 27828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [term 1 LEADER]: Becoming Leader. State: Replica: 0717054e3f0d4411b538aebeeeee5d1f, State: Running, Role: LEADER
I20260929 16:34:23.610435 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:23.610997 27828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0717054e3f0d4411b538aebeeeee5d1f" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37409 } }
I20260929 16:34:23.613603 27830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0717054e3f0d4411b538aebeeeee5d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0717054e3f0d4411b538aebeeeee5d1f" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37409 } } }
I20260929 16:34:23.614040 27830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:23.616927 27829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0717054e3f0d4411b538aebeeeee5d1f. Latest consensus state: current_term: 1 leader_uuid: "0717054e3f0d4411b538aebeeeee5d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0717054e3f0d4411b538aebeeeee5d1f" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 37409 } } }
I20260929 16:34:23.618316 27829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:23.618870 27823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:23.623737 27832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:23.629413 27832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:23.653880 27832 catalog_manager.cc:1347] Generated new cluster ID: 09e33dc84d6c4974a220f2c405492396
I20260929 16:34:23.654024 27832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:23.669999 27826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:23.636913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49805 (local address 127.5.232.190:37409)
0929 16:34:23.637170 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:23.637180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:23.645470 (+ 8290us) server_negotiation.cc:407] Connection header received
0929 16:34:23.646417 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:23.646427 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:23.646543 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:23.646670 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:23.649481 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:23.651125 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:23.653761 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:23.654786 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:23.654898 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:23.659558 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:23.659570 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:23.659578 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:23.659643 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:23.664629 (+ 4986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:23.666133 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:23.666144 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:23.666151 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:23.666416 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:23.667210 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:23.667219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:23.667225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:23.667775 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:23.668114 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:23.669074 (+ 960us) server_negotiation.cc:299] Negotiation successful
0929 16:34:23.669400 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:34:23.676765 27832 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:23.678252 27832 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:23.687606 27832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0717054e3f0d4411b538aebeeeee5d1f: Generated new TSK 0
I20260929 16:34:23.688552 27832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:23.695365 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27756
I20260929 16:34:23.696828 27818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:23.795128 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 27756
W20260929 16:34:23.813077 27741 connection.cc:535] client connection to 127.5.232.190:37409 recv error: Network error: failed to read from TLS socket (remote: 127.5.232.190:37409): Connection reset by peer (error 104)
2026-09-29T16:34:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2052 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:23 dist-test-slave-dssx krb5kdc[27856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:23 dist-test-slave-dssx krb5kdc[27856](info): set up 2 sockets
Sep 29 16:34:23 dist-test-slave-dssx krb5kdc[27856](info): commencing operation
krb5kdc: starting...
W20260929 16:34:24.905982 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:25 dist-test-slave-dssx krb5kdc[27856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:34:25.174731 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:33395
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:33395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:25.528083 27869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:25.528403 27869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:25.528498 27869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:25.536140 27869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:25.536278 27869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:25.536331 27869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:25.536396 27869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:25.536440 27869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:25.546708 27869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:33395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:25.548044 27869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:25.549420 27869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:25.561954 27877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:25.562109 27874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:25.562127 27875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:25.563367 27869 server_base.cc:1034] running on GCE node
I20260929 16:34:25.564266 27869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:25.565804 27869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:25.567026 27869 hybrid_clock.cc:648] HybridClock initialized: now 1790699665566979 us; error 36 us; skew 500 ppm
I20260929 16:34:25.567409 27869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:25 dist-test-slave-dssx krb5kdc[27856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699665, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:25.573874 27869 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:34:25.575539 27869 webserver.cc:460] Webserver started at http://127.5.232.190:34335/ using document root <none> and password file <none>
I20260929 16:34:25.576236 27869 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:25.576340 27869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:25.576604 27869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:25.579720 27869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "7062f9652ced49b88099e66428cc92d7"
format_stamp: "Formatted at 2026-09-29 16:34:25 on dist-test-slave-dssx"
server_key: "7d6b5de33f86e225b6adeb53294a66f4"
server_key_iv: "7855cfefb301ade2bc88954cc710d6a4"
server_key_version: "encryptionkey@0"
I20260929 16:34:25.580497 27869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "7062f9652ced49b88099e66428cc92d7"
format_stamp: "Formatted at 2026-09-29 16:34:25 on dist-test-slave-dssx"
server_key: "7d6b5de33f86e225b6adeb53294a66f4"
server_key_iv: "7855cfefb301ade2bc88954cc710d6a4"
server_key_version: "encryptionkey@0"
I20260929 16:34:25.586050 27869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:34:25.590198 27884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:25.591709 27869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:25.591895 27869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "7062f9652ced49b88099e66428cc92d7"
format_stamp: "Formatted at 2026-09-29 16:34:25 on dist-test-slave-dssx"
server_key: "7d6b5de33f86e225b6adeb53294a66f4"
server_key_iv: "7855cfefb301ade2bc88954cc710d6a4"
server_key_version: "encryptionkey@0"
I20260929 16:34:25.592121 27869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:25.614765 27869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:25.619043 27869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:25.619372 27869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:25.639386 27869 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:33395
I20260929 16:34:25.639426 27935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:33395 every 8 connection(s)
I20260929 16:34:25.640888 27869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:25.641817 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 27869
I20260929 16:34:25.642046 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:25.642613 6050 external_mini_cluster.cc:1428] Setting key 574177c915acc80f9c87c17903604cde
I20260929 16:34:25.646119 27936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:25.656944 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:25.644738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47387 (local address 127.5.232.190:33395)
0929 16:34:25.645593 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:25.645609 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:25.645662 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:34:25.647480 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:25.647506 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:25.647938 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:25.648388 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:25.649010 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:25.650488 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:25.651929 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:25.652667 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:25.652883 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:25.654361 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:25.654369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:25.654390 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:25.655028 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:25.655117 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:25.655149 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:25.655363 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:25.655683 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 16:34:25.655882 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":473,"threads_started":1}
I20260929 16:34:25.658155 27936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7: Bootstrap starting.
I20260929 16:34:25.661365 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:25.661967 27936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:25.663076 27936 log.cc:824] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:25.666744 27936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7: No bootstrap required, opened a new log
I20260929 16:34:25.674218 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:25.666112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45029 (local address 127.5.232.190:33395)
0929 16:34:25.666319 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:25.666326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:25.667463 (+ 1137us) server_negotiation.cc:407] Connection header received
0929 16:34:25.667684 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:25.667688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:25.667752 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:25.667843 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:25.668502 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:25.669303 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:25.671333 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:25.671772 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:25.671866 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:25.673228 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:25.673233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:25.673237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:25.673258 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:25.673303 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:25.673321 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:25.673505 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:25.673743 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 16:34:25.673847 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:34:25.677199 27936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7062f9652ced49b88099e66428cc92d7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33395 } }
I20260929 16:34:25.677834 27936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:25.678004 27936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7062f9652ced49b88099e66428cc92d7, State: Initialized, Role: FOLLOWER
I20260929 16:34:25.678674 27936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7062f9652ced49b88099e66428cc92d7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33395 } }
I20260929 16:34:25.678938 27936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:25.679085 27936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:25.679265 27936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:25.681659 27936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7062f9652ced49b88099e66428cc92d7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33395 } }
I20260929 16:34:25.682313 27936 leader_election.cc:302] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7062f9652ced49b88099e66428cc92d7; no voters:
I20260929 16:34:25.683040 27936 leader_election.cc:288] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:25.683176 27946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:25.683828 27946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [term 1 LEADER]: Becoming Leader. State: Replica: 7062f9652ced49b88099e66428cc92d7, State: Running, Role: LEADER
I20260929 16:34:25.684540 27946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7062f9652ced49b88099e66428cc92d7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33395 } }
I20260929 16:34:25.685554 27936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:25.687613 27947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7062f9652ced49b88099e66428cc92d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7062f9652ced49b88099e66428cc92d7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33395 } } }
I20260929 16:34:25.688054 27947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:25.688560 27948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7062f9652ced49b88099e66428cc92d7. Latest consensus state: current_term: 1 leader_uuid: "7062f9652ced49b88099e66428cc92d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7062f9652ced49b88099e66428cc92d7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 33395 } } }
I20260929 16:34:25.688865 27948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:25.690352 27952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:25.696255 27952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:25.709417 27952 catalog_manager.cc:1347] Generated new cluster ID: 6ad40f4eb60a4b5380c8699d24068456
I20260929 16:34:25.709563 27952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:25.735561 27952 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:25.737697 27952 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:25.747049 27952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7062f9652ced49b88099e66428cc92d7: Generated new TSK 0
I20260929 16:34:25.748281 27952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:25.768573 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27869
W20260929 16:34:25.770015 27931 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47387
W20260929 16:34:25.770911 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27869: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:25.771106 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 27869
W20260929 16:34:25.785924 27854 connection.cc:535] client connection to 127.5.232.190:33395 recv error: Network error: failed to read from TLS socket (remote: 127.5.232.190:33395): Connection reset by peer (error 104)
2026-09-29T16:34:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:25 dist-test-slave-dssx krb5kdc[27967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:25 dist-test-slave-dssx krb5kdc[27967](info): set up 2 sockets
Sep 29 16:34:25 dist-test-slave-dssx krb5kdc[27967](info): commencing operation
krb5kdc: starting...
W20260929 16:34:26.893915 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.008s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:27 dist-test-slave-dssx krb5kdc[27967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:34:27.162189 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40717
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:39663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:40717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:27.506711 27980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:27.507033 27980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:27.507268 27980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:27.514716 27980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:27.514853 27980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:27.514956 27980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:27.515056 27980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:27.515148 27980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:27.526916 27980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:39663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:40717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:40717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:27.528374 27980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:27.529861 27980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:27.542192 27985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:27.542232 27988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:27.542131 27986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:27.543113 27980 server_base.cc:1034] running on GCE node
I20260929 16:34:27.543898 27980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:27.545547 27980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:27.546769 27980 hybrid_clock.cc:648] HybridClock initialized: now 1790699667546730 us; error 33 us; skew 500 ppm
I20260929 16:34:27.547187 27980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:27 dist-test-slave-dssx krb5kdc[27967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699667, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:27.550565 27980 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:34:27.552088 27980 webserver.cc:460] Webserver started at http://127.5.232.190:37589/ using document root <none> and password file <none>
I20260929 16:34:27.552778 27980 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:27.552881 27980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:27.553151 27980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:27.556352 27980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc"
format_stamp: "Formatted at 2026-09-29 16:34:27 on dist-test-slave-dssx"
server_key: "e264ffd24a04bb3f8078564f9b3f6fd8"
server_key_iv: "53a50d566729ddce79958aed18434b20"
server_key_version: "encryptionkey@0"
I20260929 16:34:27.557154 27980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc"
format_stamp: "Formatted at 2026-09-29 16:34:27 on dist-test-slave-dssx"
server_key: "e264ffd24a04bb3f8078564f9b3f6fd8"
server_key_iv: "53a50d566729ddce79958aed18434b20"
server_key_version: "encryptionkey@0"
I20260929 16:34:27.562780 27980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:34:27.566509 27995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:27.568068 27980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:27.568274 27980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc"
format_stamp: "Formatted at 2026-09-29 16:34:27 on dist-test-slave-dssx"
server_key: "e264ffd24a04bb3f8078564f9b3f6fd8"
server_key_iv: "53a50d566729ddce79958aed18434b20"
server_key_version: "encryptionkey@0"
I20260929 16:34:27.568496 27980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:27.574687 27980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:27.579103 27980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:27.579424 27980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:27.599001 27980 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:40717
I20260929 16:34:27.599025 28046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:40717 every 8 connection(s)
I20260929 16:34:27.600519 27980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:27.604417 28047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:27.608700 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 27980
I20260929 16:34:27.608933 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:27.609447 6050 external_mini_cluster.cc:1428] Setting key c84ed5f8602e9115aa527c65b11545f2
I20260929 16:34:27.616166 28047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc: Bootstrap starting.
I20260929 16:34:27.618330 28050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:27.611303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49921 (local address 127.5.232.190:40717)
0929 16:34:27.612136 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:27.612149 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:27.612196 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:34:27.614651 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:27.614681 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:27.615069 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:27.615495 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:27.616589 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:27.616596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:27.616610 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:27.616624 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:27.616688 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:27.616710 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:27.616949 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:27.617098 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:34:27.617292 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":357,"threads_started":1}
I20260929 16:34:27.620476 28047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:27.621977 28047 log.cc:824] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:27.622545 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:27.625113 28047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc: No bootstrap required, opened a new log
I20260929 16:34:27.630074 28050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:27.627372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40153 (local address 127.5.232.190:40717)
0929 16:34:27.627592 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:27.627600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:27.627860 (+ 260us) server_negotiation.cc:407] Connection header received
0929 16:34:27.628080 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:27.628085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:27.628153 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:27.628277 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:27.629315 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:27.629320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:27.629325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:27.629350 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:27.629448 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:27.629470 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:27.629604 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:27.629696 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 16:34:27.629759 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:34:27.634339 28047 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40717 } }
I20260929 16:34:27.634764 28047 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:27.634897 28047 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ac8c0e7d7c4f149d1ac0c0b21acccc, State: Initialized, Role: FOLLOWER
I20260929 16:34:27.635505 28047 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40717 } }
I20260929 16:34:27.635771 28047 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:27.635928 28047 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:27.636113 28047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:27.638501 28047 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40717 } }
I20260929 16:34:27.639102 28047 leader_election.cc:302] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ac8c0e7d7c4f149d1ac0c0b21acccc; no voters:
I20260929 16:34:27.639715 28047 leader_election.cc:288] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:27.639830 28057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:27.640573 28057 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [term 1 LEADER]: Becoming Leader. State: Replica: b5ac8c0e7d7c4f149d1ac0c0b21acccc, State: Running, Role: LEADER
I20260929 16:34:27.641160 28057 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40717 } }
I20260929 16:34:27.641999 28047 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:27.644083 28059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ac8c0e7d7c4f149d1ac0c0b21acccc. Latest consensus state: current_term: 1 leader_uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40717 } } }
I20260929 16:34:27.644564 28059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:27.644065 28058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac8c0e7d7c4f149d1ac0c0b21acccc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 40717 } } }
I20260929 16:34:27.644999 28058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:27.647753 28063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:27.653399 28063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:27.663632 28063 catalog_manager.cc:1347] Generated new cluster ID: 21b45bda67bb4320b977cd660f74878e
I20260929 16:34:27.663763 28063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:27.677922 28063 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:27.678511 28063 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:27.684913 28063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5ac8c0e7d7c4f149d1ac0c0b21acccc: Generated new TSK 0
I20260929 16:34:27.686000 28063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:27.693799 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27980
W20260929 16:34:27.695068 28042 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49921
W20260929 16:34:27.695925 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 27980: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:27.696120 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 27980
2026-09-29T16:34:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:27 dist-test-slave-dssx krb5kdc[28078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:27 dist-test-slave-dssx krb5kdc[28078](info): set up 2 sockets
Sep 29 16:34:27 dist-test-slave-dssx krb5kdc[28078](info): commencing operation
krb5kdc: starting...
W20260929 16:34:28.805851 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:28 dist-test-slave-dssx krb5kdc[28078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:34:29.066856 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:45769
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:45769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:29.409665 28091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:29.410012 28091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:29.410109 28091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:29.417563 28091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:29.417685 28091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:29.417732 28091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:29.417793 28091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:29.417829 28091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:29.427860 28091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:45769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:45769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:29.429193 28091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:29.430545 28091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:29.442895 28097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:29.442987 28096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:29.442931 28099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:29.443850 28091 server_base.cc:1034] running on GCE node
I20260929 16:34:29.444607 28091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:29.446182 28091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:29.447433 28091 hybrid_clock.cc:648] HybridClock initialized: now 1790699669447400 us; error 47 us; skew 500 ppm
I20260929 16:34:29.447809 28091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:29 dist-test-slave-dssx krb5kdc[28078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:29.451108 28091 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:34:29.452803 28091 webserver.cc:460] Webserver started at http://127.5.232.190:32803/ using document root <none> and password file <none>
I20260929 16:34:29.453603 28091 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:29.453759 28091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:29.454087 28091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:29.457177 28091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "7f97902dcdcf4f29a254b01466c0adc7"
format_stamp: "Formatted at 2026-09-29 16:34:29 on dist-test-slave-dssx"
server_key: "7ac60a043868152277af3e7e434679c3"
server_key_iv: "b3ada7651e2fed6ecd4c6a0de9195918"
server_key_version: "encryptionkey@0"
I20260929 16:34:29.458074 28091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "7f97902dcdcf4f29a254b01466c0adc7"
format_stamp: "Formatted at 2026-09-29 16:34:29 on dist-test-slave-dssx"
server_key: "7ac60a043868152277af3e7e434679c3"
server_key_iv: "b3ada7651e2fed6ecd4c6a0de9195918"
server_key_version: "encryptionkey@0"
I20260929 16:34:29.463529 28091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:29.467262 28106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:29.468653 28091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:34:29.468866 28091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "7f97902dcdcf4f29a254b01466c0adc7"
format_stamp: "Formatted at 2026-09-29 16:34:29 on dist-test-slave-dssx"
server_key: "7ac60a043868152277af3e7e434679c3"
server_key_iv: "b3ada7651e2fed6ecd4c6a0de9195918"
server_key_version: "encryptionkey@0"
I20260929 16:34:29.469089 28091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:29.490163 28091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:29.494500 28091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:29.494822 28091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:29.514683 28091 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:45769
I20260929 16:34:29.514703 28157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:45769 every 8 connection(s)
I20260929 16:34:29.516201 28091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:29.520248 28158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:29.523788 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 28091
I20260929 16:34:29.523998 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:29.524434 6050 external_mini_cluster.cc:1428] Setting key 50ec202e12423f085d851454696c53e9
I20260929 16:34:29.533732 28158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7: Bootstrap starting.
I20260929 16:34:29.537144 28158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:29.538305 28158 log.cc:824] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:29.542306 28158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7: No bootstrap required, opened a new log
Sep 29 16:34:29 dist-test-slave-dssx krb5kdc[28078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:34:29.550385 28158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f97902dcdcf4f29a254b01466c0adc7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45769 } }
I20260929 16:34:29.550908 28158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:29.551054 28158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f97902dcdcf4f29a254b01466c0adc7, State: Initialized, Role: FOLLOWER
I20260929 16:34:29.551743 28158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f97902dcdcf4f29a254b01466c0adc7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45769 } }
I20260929 16:34:29.552027 28158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:29.552070 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:29.526355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44283 (local address 127.5.232.190:45769)
0929 16:34:29.530654 (+ 4299us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:29.530667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:29.530713 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:34:29.532626 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:29.532653 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:29.533100 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:29.533549 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:29.535060 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:29.536379 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:29.537910 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:29.538711 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:29.538947 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:29.543952 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:29.543964 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:29.543980 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:29.544023 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:29.547422 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:29.548212 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:29.548220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:29.548227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:29.548414 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:29.548866 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:29.548871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:29.548873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:29.549551 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:29.549872 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:29.550387 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 16:34:29.550884 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4022,"thread_start_us":3779,"threads_started":1}
I20260929 16:34:29.552183 28158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:29.552393 28158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:29.555452 28158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f97902dcdcf4f29a254b01466c0adc7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45769 } }
I20260929 16:34:29.556079 28158 leader_election.cc:302] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f97902dcdcf4f29a254b01466c0adc7; no voters:
I20260929 16:34:29.556280 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:29.556761 28158 leader_election.cc:288] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:29.556892 28163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:29.559005 28158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:29.564059 28163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [term 1 LEADER]: Becoming Leader. State: Replica: 7f97902dcdcf4f29a254b01466c0adc7, State: Running, Role: LEADER
I20260929 16:34:29.564914 28163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f97902dcdcf4f29a254b01466c0adc7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45769 } }
I20260929 16:34:29.568782 28164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f97902dcdcf4f29a254b01466c0adc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f97902dcdcf4f29a254b01466c0adc7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45769 } } }
I20260929 16:34:29.569201 28164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:29.569936 28165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f97902dcdcf4f29a254b01466c0adc7. Latest consensus state: current_term: 1 leader_uuid: "7f97902dcdcf4f29a254b01466c0adc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f97902dcdcf4f29a254b01466c0adc7" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45769 } } }
I20260929 16:34:29.570250 28165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7 [sys.catalog]: This master's current role is: LEADER
W20260929 16:34:29.575533 28183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:34:29.575681 28183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:34:29.575850 28182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:29.578171 28182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:29.584059 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:29.566745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60839 (local address 127.5.232.190:45769)
0929 16:34:29.566998 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:29.567007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:29.567222 (+ 215us) server_negotiation.cc:407] Connection header received
0929 16:34:29.567511 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:29.567516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:29.567613 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:29.567747 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:29.570320 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:29.571523 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:29.573033 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:29.573758 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:29.573919 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:29.576768 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:29.576780 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:29.576786 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:29.576830 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:29.581001 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:29.581889 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:29.581896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:29.581900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:29.582041 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:29.582454 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:29.582461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:29.582469 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:29.582798 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:29.583059 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:29.583422 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 16:34:29.583658 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:34:29.587797 28182 catalog_manager.cc:1347] Generated new cluster ID: 2bb185c66e474e2c94873dbab8ddf470
I20260929 16:34:29.587990 28182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:29.634948 28182 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:29.636375 28182 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:29.649655 28182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f97902dcdcf4f29a254b01466c0adc7: Generated new TSK 0
I20260929 16:34:29.650624 28182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:29.656906 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28091
I20260929 16:34:29.658239 28153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:29.758154 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 28091
2026-09-29T16:34:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:29 dist-test-slave-dssx krb5kdc[28190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:29 dist-test-slave-dssx krb5kdc[28190](info): set up 2 sockets
Sep 29 16:34:29 dist-test-slave-dssx krb5kdc[28190](info): commencing operation
krb5kdc: starting...
W20260929 16:34:30.861979 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:31 dist-test-slave-dssx krb5kdc[28190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:34:31.127921 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:45653
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:45653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:31.469758 28203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:31.470104 28203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:31.470284 28203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:31.477710 28203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:31.477836 28203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:31.477964 28203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:31.478050 28203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:31.478114 28203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:31.488309 28203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:45653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:45653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:31.489760 28203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:31.491191 28203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:31.503741 28208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:31.503795 28209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:31.503772 28211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:31.504755 28203 server_base.cc:1034] running on GCE node
I20260929 16:34:31.505700 28203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:31.507300 28203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:31.508569 28203 hybrid_clock.cc:648] HybridClock initialized: now 1790699671508528 us; error 68 us; skew 500 ppm
I20260929 16:34:31.508977 28203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:31 dist-test-slave-dssx krb5kdc[28190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699671, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:31.515846 28203 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:34:31.517625 28203 webserver.cc:460] Webserver started at http://127.5.232.190:42931/ using document root <none> and password file <none>
I20260929 16:34:31.518380 28203 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:31.518550 28203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:31.518873 28203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:31.522184 28203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d"
format_stamp: "Formatted at 2026-09-29 16:34:31 on dist-test-slave-dssx"
server_key: "49673f9dc65b8442e743873814f02b2c"
server_key_iv: "5f1be1fcdb26cfe96554edb1a2a17d5c"
server_key_version: "encryptionkey@0"
I20260929 16:34:31.523223 28203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d"
format_stamp: "Formatted at 2026-09-29 16:34:31 on dist-test-slave-dssx"
server_key: "49673f9dc65b8442e743873814f02b2c"
server_key_iv: "5f1be1fcdb26cfe96554edb1a2a17d5c"
server_key_version: "encryptionkey@0"
I20260929 16:34:31.529032 28203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:34:31.532913 28218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:31.534766 28203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:31.534962 28203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d"
format_stamp: "Formatted at 2026-09-29 16:34:31 on dist-test-slave-dssx"
server_key: "49673f9dc65b8442e743873814f02b2c"
server_key_iv: "5f1be1fcdb26cfe96554edb1a2a17d5c"
server_key_version: "encryptionkey@0"
I20260929 16:34:31.535259 28203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:31.550621 28203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:31.555430 28203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:31.555755 28203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:31.575183 28203 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:45653
I20260929 16:34:31.575202 28269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:45653 every 8 connection(s)
I20260929 16:34:31.576705 28203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:31.580619 28270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:31.584141 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 28203
I20260929 16:34:31.584322 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:31.584841 6050 external_mini_cluster.cc:1428] Setting key 634d15b7ec71ae68cd69ad123eda0106
I20260929 16:34:31.593062 28270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d: Bootstrap starting.
I20260929 16:34:31.596475 28270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:31.597771 28270 log.cc:824] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:31.599409 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:31.586863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57627 (local address 127.5.232.190:45653)
0929 16:34:31.587657 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:31.587672 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:31.587715 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:34:31.589556 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:31.589584 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:31.590074 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:31.590532 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:31.591154 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:31.592476 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:31.594018 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:31.594911 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:31.595185 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:31.596494 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:31.596502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:31.596518 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:31.597177 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:31.597260 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:31.597288 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:31.597454 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:31.597716 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 16:34:31.598019 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":344,"threads_started":1}
I20260929 16:34:31.600983 28270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d: No bootstrap required, opened a new log
I20260929 16:34:31.603682 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:31.609933 28270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45653 } }
I20260929 16:34:31.610347 28270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:31.610484 28270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b2084e7b1c4f8d8bc57ae80a4b144d, State: Initialized, Role: FOLLOWER
I20260929 16:34:31.611171 28270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45653 } }
I20260929 16:34:31.611459 28270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:31.611617 28270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:31.611804 28270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:31.614403 28270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45653 } }
I20260929 16:34:31.614887 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:31.608050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50437 (local address 127.5.232.190:45653)
0929 16:34:31.608216 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:31.608220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:31.608727 (+ 507us) server_negotiation.cc:407] Connection header received
0929 16:34:31.608955 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:31.608960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:31.609036 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:31.609145 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:31.609742 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:31.610628 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:31.612144 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:31.612694 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:31.612875 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:31.614112 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:31.614118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:31.614121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:31.614141 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:31.614184 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:31.614200 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:31.614302 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:31.614458 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:34:31.614558 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":56}
I20260929 16:34:31.615134 28270 leader_election.cc:302] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39b2084e7b1c4f8d8bc57ae80a4b144d; no voters:
I20260929 16:34:31.615832 28270 leader_election.cc:288] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:31.615981 28280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:31.617022 28280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [term 1 LEADER]: Becoming Leader. State: Replica: 39b2084e7b1c4f8d8bc57ae80a4b144d, State: Running, Role: LEADER
I20260929 16:34:31.617643 28280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45653 } }
I20260929 16:34:31.618207 28270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:31.620532 28281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45653 } } }
I20260929 16:34:31.621023 28281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:31.622637 28282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39b2084e7b1c4f8d8bc57ae80a4b144d. Latest consensus state: current_term: 1 leader_uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39b2084e7b1c4f8d8bc57ae80a4b144d" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 45653 } } }
I20260929 16:34:31.623360 28282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:31.626238 28289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:31.631850 28289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:31.641830 28289 catalog_manager.cc:1347] Generated new cluster ID: 415c0d684a9a41d6932d911f3fc2f99b
I20260929 16:34:31.641976 28289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:31.655273 28289 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:31.657250 28289 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:31.680891 28289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39b2084e7b1c4f8d8bc57ae80a4b144d: Generated new TSK 0
I20260929 16:34:31.681934 28289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:31.696010 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28203
W20260929 16:34:31.697413 28265 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57627
W20260929 16:34:31.698240 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:31.698391 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 28203
2026-09-29T16:34:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:31 dist-test-slave-dssx krb5kdc[28301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:31 dist-test-slave-dssx krb5kdc[28301](info): set up 2 sockets
Sep 29 16:34:31 dist-test-slave-dssx krb5kdc[28301](info): commencing operation
krb5kdc: starting...
W20260929 16:34:32.801991 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:32 dist-test-slave-dssx krb5kdc[28301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:34:33.066561 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:36419
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:36419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:33.413275 28314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:33.413666 28314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:33.413834 28314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:33.421224 28314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:33.421343 28314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:33.421463 28314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:33.421573 28314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:33.421657 28314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:33.431813 28314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:36419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:33.433202 28314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:33.434787 28314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:33.448133 28319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:33.452649 28322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:33.452715 28320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:33.452775 28314 server_base.cc:1034] running on GCE node
I20260929 16:34:33.453750 28314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:33.455228 28314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:33.456467 28314 hybrid_clock.cc:648] HybridClock initialized: now 1790699673456423 us; error 37 us; skew 500 ppm
I20260929 16:34:33.456827 28314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:33 dist-test-slave-dssx krb5kdc[28301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699673, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:33.460165 28314 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:34:33.461758 28314 webserver.cc:460] Webserver started at http://127.5.232.190:44917/ using document root <none> and password file <none>
I20260929 16:34:33.462476 28314 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:33.462623 28314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:33.462931 28314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:33.466090 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "b364800f4af349969287a64e1049ffdc"
format_stamp: "Formatted at 2026-09-29 16:34:33 on dist-test-slave-dssx"
server_key: "a9380d099997ae0fde56784e42e9e3d7"
server_key_iv: "1f2872ec51eed390fb2046b6b0ad8d3a"
server_key_version: "encryptionkey@0"
I20260929 16:34:33.466869 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "b364800f4af349969287a64e1049ffdc"
format_stamp: "Formatted at 2026-09-29 16:34:33 on dist-test-slave-dssx"
server_key: "a9380d099997ae0fde56784e42e9e3d7"
server_key_iv: "1f2872ec51eed390fb2046b6b0ad8d3a"
server_key_version: "encryptionkey@0"
I20260929 16:34:33.472151 28314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:33.476210 28329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:33.477716 28314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:33.477910 28314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "b364800f4af349969287a64e1049ffdc"
format_stamp: "Formatted at 2026-09-29 16:34:33 on dist-test-slave-dssx"
server_key: "a9380d099997ae0fde56784e42e9e3d7"
server_key_iv: "1f2872ec51eed390fb2046b6b0ad8d3a"
server_key_version: "encryptionkey@0"
I20260929 16:34:33.478135 28314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:33.484055 28314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:33.488648 28314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:33.488965 28314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:33.507455 28314 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:36419
I20260929 16:34:33.507486 28380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:36419 every 8 connection(s)
I20260929 16:34:33.508905 28314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:33.512686 28381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:33.513605 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 28314
I20260929 16:34:33.513813 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:33.514359 6050 external_mini_cluster.cc:1428] Setting key 83122723b3bd8425f47c526468c3c9fd
I20260929 16:34:33.522521 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.516204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43329 (local address 127.5.232.190:36419)
0929 16:34:33.517026 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:33.517037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:33.517085 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:34:33.519079 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:33.519103 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:33.519477 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:33.519840 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:33.520905 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:33.520912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:33.520925 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:33.520940 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:33.521000 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:33.521022 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:33.521216 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:33.521320 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 16:34:33.521567 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":373,"threads_started":1}
I20260929 16:34:33.526136 28381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc: Bootstrap starting.
I20260929 16:34:33.526373 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:33.529286 28381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:33.530391 28381 log.cc:824] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:33.533280 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:33.530638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46365 (local address 127.5.232.190:36419)
0929 16:34:33.530869 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:33.530879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:33.531366 (+ 487us) server_negotiation.cc:407] Connection header received
0929 16:34:33.531494 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:33.531499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:33.531569 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:33.531689 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:33.532590 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:33.532594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:33.532597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:34:33.532613 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:33.532649 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:34:33.532665 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:33.532753 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:33.532894 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 16:34:33.532941 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:34:33.533974 28381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc: No bootstrap required, opened a new log
I20260929 16:34:33.541682 28381 raft_consensus.cc:348] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b364800f4af349969287a64e1049ffdc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36419 } }
I20260929 16:34:33.542289 28381 raft_consensus.cc:374] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:34:33.542407 28381 raft_consensus.cc:729] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b364800f4af349969287a64e1049ffdc, State: Initialized, Role: FOLLOWER
I20260929 16:34:33.543109 28381 consensus_queue.cc:260] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b364800f4af349969287a64e1049ffdc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36419 } }
I20260929 16:34:33.543423 28381 raft_consensus.cc:388] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:34:33.543599 28381 raft_consensus.cc:482] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:34:33.543790 28381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:33.546655 28381 raft_consensus.cc:504] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b364800f4af349969287a64e1049ffdc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36419 } }
I20260929 16:34:33.547307 28381 leader_election.cc:302] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b364800f4af349969287a64e1049ffdc; no voters:
I20260929 16:34:33.547909 28381 leader_election.cc:288] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:34:33.548084 28391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:33.548794 28391 raft_consensus.cc:686] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [term 1 LEADER]: Becoming Leader. State: Replica: b364800f4af349969287a64e1049ffdc, State: Running, Role: LEADER
I20260929 16:34:33.549365 28391 consensus_queue.cc:237] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b364800f4af349969287a64e1049ffdc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36419 } }
I20260929 16:34:33.550187 28381 sys_catalog.cc:564] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:33.552758 28392 sys_catalog.cc:455] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b364800f4af349969287a64e1049ffdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b364800f4af349969287a64e1049ffdc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36419 } } }
I20260929 16:34:33.553256 28392 sys_catalog.cc:458] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:33.553480 28393 sys_catalog.cc:455] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b364800f4af349969287a64e1049ffdc. Latest consensus state: current_term: 1 leader_uuid: "b364800f4af349969287a64e1049ffdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b364800f4af349969287a64e1049ffdc" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 36419 } } }
I20260929 16:34:33.553766 28393 sys_catalog.cc:458] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc [sys.catalog]: This master's current role is: LEADER
W20260929 16:34:33.560788 28406 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:34:33.560913 28406 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:34:33.561076 28401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:33.562819 28401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:33.571748 28401 catalog_manager.cc:1347] Generated new cluster ID: 5113847007314d76bdfef758480e1840
I20260929 16:34:33.571909 28401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:33.598735 28401 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:33.599277 28401 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:33.616248 28401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b364800f4af349969287a64e1049ffdc: Generated new TSK 0
I20260929 16:34:33.617406 28401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:33.623281 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28314
W20260929 16:34:33.625212 28376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43329
W20260929 16:34:33.626000 6050 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28314: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:34:33.626168 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 28314
2026-09-29T16:34:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1924 ms)
[----------] 18 tests from AuthTokenIssuingTest (35482 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:33 dist-test-slave-dssx krb5kdc[28412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:33 dist-test-slave-dssx krb5kdc[28412](info): set up 2 sockets
Sep 29 16:34:33 dist-test-slave-dssx krb5kdc[28412](info): commencing operation
krb5kdc: starting...
W20260929 16:34:34.746223 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:34 dist-test-slave-dssx krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:34:35.015254 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:44059
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:44059,127.5.232.189:44389,127.5.232.188:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:35.355655 28425 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:34:35.356012 28425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:35.356091 28425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:35.356173 28425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:35.363685 28425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:35.363785 28425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:35.363842 28425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:35.363893 28425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:35.363955 28425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:35.374074 28425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:44059,127.5.232.189:44389,127.5.232.188:33425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:35.375422 28425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:35.376709 28425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:35.388639 28430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:35.388666 28433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:35.388616 28431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:35.389621 28425 server_base.cc:1034] running on GCE node
I20260929 16:34:35.390442 28425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:35.391862 28425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:35.393092 28425 hybrid_clock.cc:648] HybridClock initialized: now 1790699675393033 us; error 30 us; skew 500 ppm
I20260929 16:34:35.393507 28425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:35 dist-test-slave-dssx krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:35.396785 28425 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:34:35.398416 28425 webserver.cc:460] Webserver started at http://127.5.232.190:33115/ using document root <none> and password file <none>
I20260929 16:34:35.399149 28425 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:35.399286 28425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:35.399623 28425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:35.402747 28425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "c8a1086e249744edaea1f9895c716097"
format_stamp: "Formatted at 2026-09-29 16:34:35 on dist-test-slave-dssx"
server_key: "8a43e0ac0bea3a7f7cdc84d890c93d99"
server_key_iv: "ac8f91ac48a01c636e88225935b9f3e4"
server_key_version: "encryptionkey@0"
I20260929 16:34:35.403568 28425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "c8a1086e249744edaea1f9895c716097"
format_stamp: "Formatted at 2026-09-29 16:34:35 on dist-test-slave-dssx"
server_key: "8a43e0ac0bea3a7f7cdc84d890c93d99"
server_key_iv: "ac8f91ac48a01c636e88225935b9f3e4"
server_key_version: "encryptionkey@0"
I20260929 16:34:35.408986 28425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:34:35.412681 28440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:35.414261 28425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:34:35.414467 28425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "c8a1086e249744edaea1f9895c716097"
format_stamp: "Formatted at 2026-09-29 16:34:35 on dist-test-slave-dssx"
server_key: "8a43e0ac0bea3a7f7cdc84d890c93d99"
server_key_iv: "ac8f91ac48a01c636e88225935b9f3e4"
server_key_version: "encryptionkey@0"
I20260929 16:34:35.414724 28425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:35.437137 28425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:35.442008 28425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:35.442327 28425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:35.462152 28425 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:44059
I20260929 16:34:35.462182 28491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:44059 every 8 connection(s)
I20260929 16:34:35.463678 28425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:35.467736 28492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:35.471769 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 28425
I20260929 16:34:35.472005 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:35.472791 6050 external_mini_cluster.cc:1428] Setting key a069ca8621c0105556f6aef2bae317b3
I20260929 16:34:35.477167 28492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:35 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.189@KRBTEST.COM" created.
I20260929 16:34:35.504572 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.481524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44059 (local address 127.0.0.1:47714)
0929 16:34:35.482433 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.482496 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:35.485409 (+ 2913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:35.486447 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:35.486457 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:35.487351 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:35.487910 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:35.487944 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:35.489591 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:35.489600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:35.491605 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:35.491626 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:35.492694 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:35.492704 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:35.492963 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:35.493796 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:35.493831 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:35.496709 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:35.500293 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:35.500306 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:35.500320 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:35.500747 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:35.501200 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:35.501205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:35.501208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:35.501470 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:35.502370 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:35.502377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:35.502584 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 16:34:35.502816 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 16:34:35.503268 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":391,"threads_started":1}
I20260929 16:34:35.504606 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.483023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.5.232.190:44059)
0929 16:34:35.483820 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:35.483835 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:35.483908 (+ 73us) server_negotiation.cc:407] Connection header received
0929 16:34:35.485887 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:35.485894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:35.486170 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:35.486347 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:35.488168 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:35.489359 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:35.491807 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:35.492410 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:35.492583 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:35.496973 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:35.496981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:35.496994 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:35.497029 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:35.500131 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:35.500893 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:35.500908 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:35.500910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:35.501074 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:35.501586 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:35.501594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:35.501597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:35.502148 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:35.502416 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:35.503036 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 16:34:35.503540 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":556,"thread_start_us":368,"threads_started":1}
I20260929 16:34:35.508404 28492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:34:35.509711 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.509167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41052)
0929 16:34:35.509464 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.509581 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260929 16:34:35.510001 28442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
I20260929 16:34:35.511731 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.511297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41066)
0929 16:34:35.511511 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.511589 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 16:34:35.511937 28492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:44389: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab.
Entry for principal kudu/127.5.232.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab.
I20260929 16:34:35.534816 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.189:44389
--webserver_interface=127.5.232.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab
--principal=kudu/127.5.232.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:44059,127.5.232.189:44389,127.5.232.188:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:34:35.561297 28492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } attempt: 1
I20260929 16:34:35.562536 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.562046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41078)
0929 16:34:35.562256 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.562386 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:34:35.563575 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.563191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41094)
0929 16:34:35.563352 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.563433 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 16:34:35.563777 28492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:44389: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
I20260929 16:34:35.632100 28492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } attempt: 2
I20260929 16:34:35.633360 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.632836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41110)
0929 16:34:35.633114 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.633212 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:34:35.634546 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.634146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41124)
0929 16:34:35.634312 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.634409 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 16:34:35.634824 28492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:44389: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
I20260929 16:34:35.730778 28492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } attempt: 3
I20260929 16:34:35.732126 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.731560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41140)
0929 16:34:35.731844 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.731957 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 16:34:35.733457 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.732914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41148)
0929 16:34:35.733104 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.733215 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:34:35.733955 28492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:44389: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
I20260929 16:34:35.877188 28492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } attempt: 4
I20260929 16:34:35.878511 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.877961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41160)
0929 16:34:35.878277 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.878381 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260929 16:34:35.879567 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:35.879230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41168)
0929 16:34:35.879369 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:35.879448 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:34:35.879864 28492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:44389: Network error: Client connection negotiation failed: client connection to 127.5.232.189:44389: connect: Connection refused (error 111)
W20260929 16:34:35.885231 28499 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:34:35.885674 28499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:35.885800 28499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:35.885943 28499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:35.893497 28499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:35.893625 28499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:35.893724 28499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:35.893821 28499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:35.893909 28499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:35.904127 28499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:44059,127.5.232.189:44389,127.5.232.188:33425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab
--principal=kudu/127.5.232.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.189:44389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.232.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:35.905653 28499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:35.907068 28499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:35.919406 28504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:35.919382 28505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:35.919687 28507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:35.920145 28499 server_base.cc:1034] running on GCE node
I20260929 16:34:35.920961 28499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:35.922436 28499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:35.923656 28499 hybrid_clock.cc:648] HybridClock initialized: now 1790699675923601 us; error 40 us; skew 500 ppm
I20260929 16:34:35.924003 28499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:35 dist-test-slave-dssx krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:35.927311 28499 init.cc:377] Logged in from keytab as kudu/127.5.232.189@KRBTEST.COM (short username kudu)
I20260929 16:34:35.928927 28499 webserver.cc:460] Webserver started at http://127.5.232.189:39921/ using document root <none> and password file <none>
I20260929 16:34:35.929772 28499 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:35.929870 28499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:35.930190 28499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:35.933140 28499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/data/instance:
uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4"
format_stamp: "Formatted at 2026-09-29 16:34:35 on dist-test-slave-dssx"
server_key: "ded426c8dc0bba984e1a6f2f908b6ed1"
server_key_iv: "0c41641af66969f1b0fffb0ffdd1abab"
server_key_version: "encryptionkey@0"
I20260929 16:34:35.933969 28499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/wal/instance:
uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4"
format_stamp: "Formatted at 2026-09-29 16:34:35 on dist-test-slave-dssx"
server_key: "ded426c8dc0bba984e1a6f2f908b6ed1"
server_key_iv: "0c41641af66969f1b0fffb0ffdd1abab"
server_key_version: "encryptionkey@0"
I20260929 16:34:35.939129 28499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:35.942679 28514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:35.944136 28499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:35.944322 28499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/wal
uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4"
format_stamp: "Formatted at 2026-09-29 16:34:35 on dist-test-slave-dssx"
server_key: "ded426c8dc0bba984e1a6f2f908b6ed1"
server_key_iv: "0c41641af66969f1b0fffb0ffdd1abab"
server_key_version: "encryptionkey@0"
I20260929 16:34:35.944568 28499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:35.962747 28499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:35.967367 28499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:35.967664 28499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:35.986733 28499 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.189:44389
I20260929 16:34:35.986752 28565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.189:44389 every 8 connection(s)
I20260929 16:34:35.988173 28499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
I20260929 16:34:35.992180 28566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:35.996452 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 28499
I20260929 16:34:35.996644 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-1/wal/instance
I20260929 16:34:35.997061 6050 external_mini_cluster.cc:1428] Setting key f4fe0ce2f62190b264304505baa144fb
I20260929 16:34:35.999229 28566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.188@KRBTEST.COM; defaulting to no policy
I20260929 16:34:36.022739 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.001963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.5.232.190:44059)
0929 16:34:36.002177 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.002181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.003647 (+ 1466us) server_negotiation.cc:407] Connection header received
0929 16:34:36.005937 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.005942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.006019 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.006134 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.008363 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.009052 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.011615 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.012055 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.012156 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.016344 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.016350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.016353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.016389 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.019496 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.020311 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.020316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.020320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.020450 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.020906 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.020911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.020915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.021223 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.021383 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.022165 (+ 782us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.022407 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:34:36.023777 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.002644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44059 (local address 127.0.0.1:47722)
0929 16:34:36.003457 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.003495 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:36.005598 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:36.006460 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:36.006476 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:36.007294 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:36.008155 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.008185 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.009237 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.009246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.011418 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.011438 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.012306 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.012315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.012672 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.013517 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:36.013552 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:36.016078 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:36.019678 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.019688 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.019712 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.020176 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.020570 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.020575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.020578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.020774 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.021354 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:36.021363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:36.021770 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 16:34:36.022050 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 16:34:36.022622 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.188@KRBTEST.COM" created.
I20260929 16:34:36.025723 28566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:34:36.042351 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.026404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41174)
0929 16:34:36.026642 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.026664 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:36.026774 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:36.027669 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:36.027681 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:36.028171 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:36.028376 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.028389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.029785 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.029793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.031363 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.031380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.032220 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.032232 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.032473 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.033107 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:36.033133 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:36.035550 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:36.039945 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.039949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.039952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.040305 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.040724 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.040727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.040729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.040871 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.041671 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:36.041674 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:36.041784 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 16:34:36.041894 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:34:36.042095 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:34:36.042841 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.026658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.5.232.189:44389)
0929 16:34:36.027236 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.027242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.027276 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:34:36.027323 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.027329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.027504 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.027627 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.028543 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.029651 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.031509 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.032069 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.032169 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.035774 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.035789 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.035810 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.035854 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.039805 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.040440 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.040452 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.040458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.040616 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.040972 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.040979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.040985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.041553 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.041762 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.042066 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.042360 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":292,"threads_started":1}
I20260929 16:34:36.045329 28566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:34:36.046654 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.046089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53366)
0929 16:34:36.046313 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.046456 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 16:34:36.046988 28517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
I20260929 16:34:36.048666 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.048290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53380)
0929 16:34:36.048481 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.048560 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 16:34:36.048877 28566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:33425: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab.
Entry for principal kudu/127.5.232.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab.
I20260929 16:34:36.060367 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.188:33425
--webserver_interface=127.5.232.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab
--principal=kudu/127.5.232.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:44059,127.5.232.189:44389,127.5.232.188:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:34:36.098219 28566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } attempt: 1
I20260929 16:34:36.099479 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.099006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53382)
0929 16:34:36.099224 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.099321 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 16:34:36.100576 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.100259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53394)
0929 16:34:36.100397 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.100470 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:34:36.100811 28566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:33425: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
I20260929 16:34:36.169064 28566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } attempt: 2
I20260929 16:34:36.170321 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.169827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53402)
0929 16:34:36.170046 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.170145 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:34:36.171504 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.171028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53406)
0929 16:34:36.171196 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.171303 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:34:36.171753 28566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:33425: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
I20260929 16:34:36.179121 28492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } attempt: 5
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:34:36.193279 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.179877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41190)
0929 16:34:36.180096 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.180118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:36.180258 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:36.180652 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:36.180659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:36.181055 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:36.181235 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.181244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.182268 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.182273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.183128 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.183141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.183963 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.183970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.184190 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.184876 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:36.184900 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:36.187436 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:36.190737 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.190743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.190748 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.191151 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.191547 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.191552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.191556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.191706 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.192264 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:36.192269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:36.192420 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 16:34:36.192599 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 16:34:36.192911 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 16:34:36.193275 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.180012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.5.232.189:44389)
0929 16:34:36.180214 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.180220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.180241 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:34:36.180408 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.180415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.180500 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.180608 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.181422 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.182123 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.183270 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.183783 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.183914 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.187665 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.187671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.187674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.187711 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.190565 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.191285 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.191292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.191295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.191427 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.191820 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.191827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.191831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.192136 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.192303 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.192694 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.192913 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:34:36.194553 28492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:34:36.195664 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.195273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53418)
0929 16:34:36.195445 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.195533 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:34:36.196610 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.196296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53426)
0929 16:34:36.196435 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.196506 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:34:36.196837 28492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:33425: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
I20260929 16:34:36.248082 28492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } attempt: 1
I20260929 16:34:36.249418 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.248888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53436)
0929 16:34:36.249087 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.249209 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:34:36.250418 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.250082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53442)
0929 16:34:36.250235 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.250311 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:34:36.250619 28492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:33425: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
I20260929 16:34:36.262980 28566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } attempt: 3
I20260929 16:34:36.264250 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.263775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53458)
0929 16:34:36.263995 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.264093 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:34:36.265439 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.264962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53470)
0929 16:34:36.265195 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.265271 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260929 16:34:36.265635 28566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:33425: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
I20260929 16:34:36.318909 28492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } attempt: 2
I20260929 16:34:36.320185 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.319599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53474)
0929 16:34:36.319850 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.319994 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 16:34:36.321276 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.320911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53476)
0929 16:34:36.321067 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.321164 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 16:34:36.321524 28492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:33425: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
W20260929 16:34:36.408150 28573 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:34:36.408583 28573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:36.408716 28573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:36.408859 28573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:34:36.408892 28566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } attempt: 4
I20260929 16:34:36.410142 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.409648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53482)
0929 16:34:36.409866 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.409959 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:34:36.411180 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.410834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53496)
0929 16:34:36.410990 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.411068 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:34:36.411391 28566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:33425: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
W20260929 16:34:36.417191 28573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:36.417331 28573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:36.417480 28573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:36.417577 28573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:36.417683 28573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:36.427750 28492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } attempt: 3
I20260929 16:34:36.428363 28573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:44059,127.5.232.189:44389,127.5.232.188:33425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab
--principal=kudu/127.5.232.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.188:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.232.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:36.428916 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.428432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53498)
0929 16:34:36.428634 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.428746 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:34:36.430030 28573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:36.430056 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.429623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53506)
0929 16:34:36.429859 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.429931 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 16:34:36.430310 28492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:33425: Network error: Client connection negotiation failed: client connection to 127.5.232.188:33425: connect: Connection refused (error 111)
I20260929 16:34:36.431510 28573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:36.444432 28578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:36.444489 28579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:36.444363 28581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:36.450778 28573 server_base.cc:1034] running on GCE node
I20260929 16:34:36.451717 28573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:36.453363 28573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:36.454696 28573 hybrid_clock.cc:648] HybridClock initialized: now 1790699676454614 us; error 58 us; skew 500 ppm
I20260929 16:34:36.455075 28573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:36.458469 28573 init.cc:377] Logged in from keytab as kudu/127.5.232.188@KRBTEST.COM (short username kudu)
I20260929 16:34:36.460063 28573 webserver.cc:460] Webserver started at http://127.5.232.188:35747/ using document root <none> and password file <none>
I20260929 16:34:36.460840 28573 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:36.460956 28573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:36.461292 28573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:36.464426 28573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/data/instance:
uuid: "d4085ec065ce4ac5a4d0f406a3d61ada"
format_stamp: "Formatted at 2026-09-29 16:34:36 on dist-test-slave-dssx"
server_key: "74fdbbbf99cde89f95976e55abe991b5"
server_key_iv: "0e8e9cc20fd9dcebae3c215d281d5354"
server_key_version: "encryptionkey@0"
I20260929 16:34:36.465277 28573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/wal/instance:
uuid: "d4085ec065ce4ac5a4d0f406a3d61ada"
format_stamp: "Formatted at 2026-09-29 16:34:36 on dist-test-slave-dssx"
server_key: "74fdbbbf99cde89f95976e55abe991b5"
server_key_iv: "0e8e9cc20fd9dcebae3c215d281d5354"
server_key_version: "encryptionkey@0"
I20260929 16:34:36.470957 28573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:34:36.474692 28588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:36.476114 28573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:34:36.476312 28573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/wal
uuid: "d4085ec065ce4ac5a4d0f406a3d61ada"
format_stamp: "Formatted at 2026-09-29 16:34:36 on dist-test-slave-dssx"
server_key: "74fdbbbf99cde89f95976e55abe991b5"
server_key_iv: "0e8e9cc20fd9dcebae3c215d281d5354"
server_key_version: "encryptionkey@0"
I20260929 16:34:36.476562 28573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:36.526799 28573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:36.531774 28573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:36.532099 28573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:36.551466 28573 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.188:33425
I20260929 16:34:36.551472 28639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.188:33425 every 8 connection(s)
I20260929 16:34:36.553040 28573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
I20260929 16:34:36.557807 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 28573
I20260929 16:34:36.557986 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699314866137-6050-0/minicluster-data/master-2/wal/instance
I20260929 16:34:36.558267 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:36.558444 6050 external_mini_cluster.cc:1428] Setting key 5ed79195b3e7c2b5bfbd447f81c3bb9f
I20260929 16:34:36.568840 28640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:34:36.584436 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.560453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.5.232.188:33425)
0929 16:34:36.561362 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.561402 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.561458 (+ 56us) server_negotiation.cc:407] Connection header received
0929 16:34:36.563525 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.563556 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.564137 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.564632 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.566239 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.568214 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.569640 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.570657 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.570852 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.575612 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.575620 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.575632 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.575678 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.579223 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.580050 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.580058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.580065 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.580283 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.580784 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.580790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.580793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.581566 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.581882 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.582418 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.582891 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":336,"threads_started":1}
I20260929 16:34:36.588073 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.571947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:44059 (local address 127.0.0.1:47734)
0929 16:34:36.572836 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.572877 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:36.573070 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:36.573499 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:36.573506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:36.574143 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:36.574391 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.574400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.575602 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.575606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.576930 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.576946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.578085 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.578096 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.578284 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.578926 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:36.578944 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:36.581203 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:36.585019 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.585053 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.585062 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.585706 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.586150 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.586155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.586158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.586350 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.586899 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:36.586906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:36.587097 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 16:34:36.587326 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 16:34:36.587666 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":528,"threads_started":1}
I20260929 16:34:36.588181 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.571876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.5.232.190:44059)
0929 16:34:36.572495 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.572503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.572978 (+ 475us) server_negotiation.cc:407] Connection header received
0929 16:34:36.573208 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.573215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.573286 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.573432 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.574551 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.575421 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.577105 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.577866 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.578036 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.581452 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.581462 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.581467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.581508 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.584874 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.585875 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.585879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.585882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.586029 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.586469 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.586472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.586475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.586767 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.586887 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.587473 (+ 586us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.587713 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":352,"threads_started":1}
I20260929 16:34:36.590425 28640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:34:36.604969 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.591141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:44389 (local address 127.0.0.1:41192)
0929 16:34:36.591302 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.591322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:36.591422 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:36.591795 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:36.591799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:36.592285 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:36.592474 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.592485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.593765 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.593769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.595067 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.595085 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.595884 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.595893 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.596123 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.596719 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:36.596743 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:36.599011 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:36.602205 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.602212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.602216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.602625 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.603155 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.603161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.603167 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.603348 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.603913 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:36.603921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:36.604101 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 16:34:36.604281 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 16:34:36.604607 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 16:34:36.604981 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.588222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.5.232.190:44059)
0929 16:34:36.588397 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.588402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.588729 (+ 327us) server_negotiation.cc:407] Connection header received
0929 16:34:36.588921 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.588926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.589018 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.589139 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.590612 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.591607 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.593275 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.593931 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.594030 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.598996 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.599003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.599007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.599048 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.602029 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.602839 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.602843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.602846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.602971 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.603520 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.603526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.603529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.603817 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.604128 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.604389 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.604619 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":720,"threads_started":1}
I20260929 16:34:36.605178 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.591271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.5.232.189:44389)
0929 16:34:36.591475 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.591482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.591501 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:34:36.591572 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.591577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.591659 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.591758 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.592652 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.593597 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.595248 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.595703 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.595802 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.599238 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.599243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.599247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.599282 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.602043 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.602761 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.602769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.602776 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.602934 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.603468 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.603475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.603478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.603773 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.603923 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.604391 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.604618 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:34:36.606211 28640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:34:36.607501 28492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } attempt: 4
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:34:36.622572 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.607007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53516)
0929 16:34:36.607184 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.607203 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:36.607302 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:36.608522 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:36.608528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:36.609025 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:36.609174 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.609183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.610574 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.610580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.611889 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.611901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.612657 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.612662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.612858 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.613547 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:36.613572 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:36.615849 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:36.619603 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.619614 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.619619 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.620075 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.620491 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.620495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.620499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.620687 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.621359 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:36.621364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:36.621576 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 16:34:36.621780 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 16:34:36.622122 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":23}
I20260929 16:34:36.622997 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.607386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.5.232.188:33425)
0929 16:34:36.607899 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.607904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.607917 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:34:36.607979 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.607983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.608059 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.608173 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.609310 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.610423 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.612028 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.612484 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.612635 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.616037 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.616042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.616045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.616077 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.619466 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.620204 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.620211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.620215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.620349 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.620825 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.620831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.620834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.621214 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.621438 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.621900 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.622311 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":421}
I20260929 16:34:36.625088 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.608076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53526)
0929 16:34:36.608263 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.608282 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:36.608376 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:36.609438 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:36.609445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:36.609938 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:36.610149 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.610160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.611435 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.611442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.612846 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.612862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.613887 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.613895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.614114 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.614777 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:36.614807 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:36.617631 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:36.621974 (+ 4343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.621979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.621984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.622543 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.623189 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.623194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.623197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.623362 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.624072 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:36.624077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:36.624234 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 16:34:36.624389 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 16:34:36.624696 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:34:36.625515 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.606854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.5.232.189:44389)
0929 16:34:36.607020 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.607025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.607042 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:34:36.607185 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.607191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.607278 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.607376 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.609044 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.610041 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.611699 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.612328 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.612449 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.618841 (+ 6392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.618850 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.618856 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.618908 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.622454 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.623313 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.623319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.623322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.623509 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.623962 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.623967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.623970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.624335 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.624597 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.624908 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.625141 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:34:36.628870 28640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada: Bootstrap starting.
I20260929 16:34:36.630303 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.608369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.5.232.188:33425)
0929 16:34:36.609059 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.609065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.609084 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:34:36.609136 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.609140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.609207 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.609364 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.610334 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.611289 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.612999 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.613720 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.613844 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.617855 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.617864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.617870 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.617909 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.621816 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.622871 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.622877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.622880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.623029 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.623512 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.623516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.623520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.623886 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.624073 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.629670 (+ 5597us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.629858 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":556,"thread_start_us":309,"threads_started":1}
I20260929 16:34:36.631162 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:36.633147 28640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:36.634608 28640 log.cc:824] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:36.634969 28492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097: Bootstrap starting.
I20260929 16:34:36.637519 28640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada: No bootstrap required, opened a new log
I20260929 16:34:36.634784 28467 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c8a1086e249744edaea1f9895c716097"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47748
I20260929 16:34:36.639360 28492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:36.640686 28492 log.cc:824] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:36.644152 28492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097: No bootstrap required, opened a new log
I20260929 16:34:36.645893 28640 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } }
I20260929 16:34:36.646311 28640 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4085ec065ce4ac5a4d0f406a3d61ada, State: Initialized, Role: FOLLOWER
I20260929 16:34:36.646786 28640 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } }
I20260929 16:34:36.647774 28650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.648280 28650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:36.649225 28640 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:36.652036 28492 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } }
I20260929 16:34:36.652450 28492 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a1086e249744edaea1f9895c716097, State: Initialized, Role: FOLLOWER
I20260929 16:34:36.652977 28492 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } }
I20260929 16:34:36.654230 28658 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.654955 28658 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:36.655673 28492 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:36.655786 28467 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:34:36.655965 28467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:36.660619 28467 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } }
I20260929 16:34:36.665254 28615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8a1086e249744edaea1f9895c716097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada"
I20260929 16:34:36.666055 28615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:36.673043 28467 leader_election.cc:288] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [CANDIDATE]: Term 1 election: Requested vote from peers 71c3df175d2e4cfb9bb2f8e7c42d38a4 (127.5.232.189:44389), d4085ec065ce4ac5a4d0f406a3d61ada (127.5.232.188:33425)
I20260929 16:34:36.674024 28615 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a1086e249744edaea1f9895c716097 in term 1.
I20260929 16:34:36.676852 28443 leader_election.cc:302] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8a1086e249744edaea1f9895c716097, d4085ec065ce4ac5a4d0f406a3d61ada; no voters:
I20260929 16:34:36.677476 28658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:36.658149 28573 master_runner.cc:186] Error getting master registration for 127.5.232.190:44059: OK, instance_id { permanent_uuid: "c8a1086e249744edaea1f9895c716097" instance_seqno: 1790699675456466 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:34:36.683038 28573 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 16:34:36.690088 28662 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:34:36.690241 28662 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:34:36.690459 28658 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [term 1 LEADER]: Becoming Leader. State: Replica: c8a1086e249744edaea1f9895c716097, State: Running, Role: LEADER
I20260929 16:34:36.691378 28658 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } }
W20260929 16:34:36.695335 28446 tablet.cc:2369] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:36.700428 28670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8a1086e249744edaea1f9895c716097. Latest consensus state: current_term: 1 leader_uuid: "c8a1086e249744edaea1f9895c716097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.701097 28670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:36.707368 28677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:36.710582 28566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } attempt: 5
Sep 29 16:34:36 dist-test-slave-dssx krb5kdc[28412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
I20260929 16:34:36.731956 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.711263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:33425 (local address 127.0.0.1:53528)
0929 16:34:36.711686 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:36.711708 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:36.711837 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:36.712780 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:36.712785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:36.713347 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:36.713622 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.713636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.715539 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.715545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.716970 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:36.716987 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.718257 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.718264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:36.718503 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.719293 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:36.719341 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:36.723015 (+ 3674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:36.728251 (+ 5236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.728258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.728264 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.728937 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.729526 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:36.729533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:36.729538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:36.729733 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:36.730644 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:36.730651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:36.730861 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 16:34:36.731047 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:34:36.731447 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260929 16:34:36.733325 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.712061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.5.232.188:33425)
0929 16:34:36.712340 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.712350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.712380 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:34:36.712469 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.712477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.712593 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.712744 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.713825 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.714993 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.717168 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.717893 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.718021 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.723343 (+ 5322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.723351 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.723358 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.723415 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.727988 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.729135 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.729143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.729149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.729346 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.729882 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.729890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.729894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.730389 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.730556 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.732546 (+ 1990us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.732856 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:34:36.738488 28677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:36.740589 28566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4: Bootstrap starting.
I20260929 16:34:36.745048 28566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:36.751013 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.720438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.5.232.190:44059)
0929 16:34:36.720634 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.720682 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.727442 (+ 6760us) server_negotiation.cc:407] Connection header received
0929 16:34:36.728022 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.728029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.728122 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.728203 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.730595 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.731715 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.734197 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.734897 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.735001 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.740984 (+ 5983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.740991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.740995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.741041 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.746310 (+ 5269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.747528 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.747538 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.747545 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.747775 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.748393 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.748401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.748407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.748889 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.749242 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.750233 (+ 991us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.750575 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:34:36.753073 28566 log.cc:824] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:36.755891 28685 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:34:36.769894 28615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [term 1 FOLLOWER]: Refusing update from remote peer c8a1086e249744edaea1f9895c716097: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:36.771092 28670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:36.764492 28425 master_runner.cc:186] Error getting master registration for 127.5.232.189:44389: OK, instance_id { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" instance_seqno: 1790699675981084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:34:36.775480 28425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:34:36.778426 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.720047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53542 (local address 127.5.232.188:33425)
0929 16:34:36.720251 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.720257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.735553 (+ 15296us) server_negotiation.cc:407] Connection header received
0929 16:34:36.735981 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.735987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.736070 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.736136 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.753699 (+ 17563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.754913 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.759734 (+ 4821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.760454 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.760562 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.768051 (+ 7489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.768059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.768064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.768117 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.772716 (+ 4599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.774484 (+ 1768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.774493 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.774498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.774707 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.775492 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.775498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.775503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.775927 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.776169 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.777208 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.777502 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:34:36.780156 28566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4: No bootstrap required, opened a new log
I20260929 16:34:36.784134 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:36.734489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.5.232.189:44389)
0929 16:34:36.734661 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:36.734667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:36.752370 (+ 17703us) server_negotiation.cc:407] Connection header received
0929 16:34:36.752864 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:36.752874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:36.752984 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:36.753086 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:36.756001 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.757142 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.771270 (+ 14128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:36.771962 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:36.772098 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:36.775894 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:36.775901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:36.775906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:36.775947 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:36.779918 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.780955 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.780961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.780964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.781158 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:36.781651 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:36.781655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:36.781658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:36.782038 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:36.782301 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:36.783440 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0929 16:34:36.783725 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 16:34:36.784338 28594 tablet.cc:2369] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:36.800130 28566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } }
I20260929 16:34:36.800940 28566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c3df175d2e4cfb9bb2f8e7c42d38a4, State: Initialized, Role: FOLLOWER
I20260929 16:34:36.801947 28566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } }
I20260929 16:34:36.810649 28650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8a1086e249744edaea1f9895c716097. Latest consensus state: current_term: 1 leader_uuid: "c8a1086e249744edaea1f9895c716097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.810973 28650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:36.820538 28686 mvcc.cc:205] Tried to move back new op lower bound from 7334705876037099520 to 7334705875735826432. Current Snapshot: MvccSnapshot[applied={T|T < 7334705876037099520}]
I20260929 16:34:36.820708 28540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:36.820820 28541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8a1086e249744edaea1f9895c716097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4"
I20260929 16:34:36.823101 28688 mvcc.cc:205] Tried to move back new op lower bound from 7334705876037099520 to 7334705875735826432. Current Snapshot: MvccSnapshot[applied={T|T < 7334705876037099520}]
I20260929 16:34:36.824528 28540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [term 1 FOLLOWER]: Refusing update from remote peer c8a1086e249744edaea1f9895c716097: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:36.827934 28541 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a1086e249744edaea1f9895c716097 in term 1.
I20260929 16:34:36.818034 28690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.828310 28690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:36.830327 28566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:36.831440 28658 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:36.835620 28658 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8a1086e249744edaea1f9895c716097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.835939 28658 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:36.836956 28677 catalog_manager.cc:1347] Generated new cluster ID: 4a13a0062cfd47fc8c9b9722bf85b46c
I20260929 16:34:36.837054 28677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:36.842973 28650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8a1086e249744edaea1f9895c716097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.843246 28650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:36.860984 28677 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:36.862936 28677 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:36.866659 28697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8a1086e249744edaea1f9895c716097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.867460 28670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8a1086e249744edaea1f9895c716097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.866866 28690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8a1086e249744edaea1f9895c716097. Latest consensus state: current_term: 1 leader_uuid: "c8a1086e249744edaea1f9895c716097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a1086e249744edaea1f9895c716097" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 44059 } } peers { permanent_uuid: "71c3df175d2e4cfb9bb2f8e7c42d38a4" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 44389 } } peers { permanent_uuid: "d4085ec065ce4ac5a4d0f406a3d61ada" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 33425 } } }
I20260929 16:34:36.867808 28670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097 [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:36.867802 28697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:36.867810 28690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:36.899962 28694 mvcc.cc:205] Tried to move back new op lower bound from 7334705876037099520 to 7334705875735826432. Current Snapshot: MvccSnapshot[applied={T|T < 7334705876037099520 or (T in {7334705876037099520})}]
I20260929 16:34:36.921795 28708 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:34:36.933671 28677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8a1086e249744edaea1f9895c716097: Generated new TSK 0
I20260929 16:34:36.936878 28677 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:34:37.041319 28643 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.027203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.5.232.188:33425)
0929 16:34:37.027420 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.027427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.039280 (+ 11853us) server_negotiation.cc:407] Connection header received
0929 16:34:37.039591 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.039598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.039711 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.039792 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.041004 (+ 1212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53544: BlockingRecv error: recv got EOF from 127.0.0.1:53544 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 16:34:37.043421 28644 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.028193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.5.232.190:44059)
0929 16:34:37.028381 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.028387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.038960 (+ 10573us) server_negotiation.cc:407] Connection header received
0929 16:34:37.040540 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.040547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.040652 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.040729 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.043115 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47760: BlockingRecv error: recv got EOF from 127.0.0.1:47760 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 16:34:37.043874 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.039543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.5.232.189:44389)
0929 16:34:37.039760 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.039766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.042293 (+ 2527us) server_negotiation.cc:407] Connection header received
0929 16:34:37.042522 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.042528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.042636 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.042708 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.043650 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41218: BlockingRecv error: recv got EOF from 127.0.0.1:41218 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 16:34:37.052662 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.050125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.5.232.189:44389)
0929 16:34:37.050488 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.050496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.050988 (+ 492us) server_negotiation.cc:407] Connection header received
0929 16:34:37.051235 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.051241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.051370 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.051485 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.052389 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41232: BlockingRecv error: recv got EOF from 127.0.0.1:41232 (error 108)
Metrics: {"server-negotiator.queue_time_us":207}
W20260929 16:34:37.061036 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.058798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.5.232.189:44389)
0929 16:34:37.058972 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.058978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.059503 (+ 525us) server_negotiation.cc:407] Connection header received
0929 16:34:37.059706 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.059711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.059804 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.059915 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.060812 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41234: BlockingRecv error: recv got EOF from 127.0.0.1:41234 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 16:34:37.078015 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.075659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.5.232.189:44389)
0929 16:34:37.075852 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.075858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.076401 (+ 543us) server_negotiation.cc:407] Connection header received
0929 16:34:37.076640 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.076646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.076747 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.076853 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.077766 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41236: BlockingRecv error: recv got EOF from 127.0.0.1:41236 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 16:34:37.087911 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.085593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.5.232.189:44389)
0929 16:34:37.085826 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.085834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.086394 (+ 560us) server_negotiation.cc:407] Connection header received
0929 16:34:37.086604 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.086610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.086721 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.086835 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.087679 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41250: BlockingRecv error: recv got EOF from 127.0.0.1:41250 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260929 16:34:37.101969 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.099652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.5.232.189:44389)
0929 16:34:37.099845 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.099852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.100420 (+ 568us) server_negotiation.cc:407] Connection header received
0929 16:34:37.100624 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.100630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.100722 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.100842 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.101694 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41258: BlockingRecv error: recv got EOF from 127.0.0.1:41258 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 16:34:37.124536 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.121938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41272 (local address 127.5.232.189:44389)
0929 16:34:37.122133 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.122141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.122717 (+ 576us) server_negotiation.cc:407] Connection header received
0929 16:34:37.122992 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.122999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.123104 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.123185 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.124299 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41272: BlockingRecv error: recv got EOF from 127.0.0.1:41272 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 16:34:37.169869 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.167277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.5.232.189:44389)
0929 16:34:37.167483 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.167490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.168071 (+ 581us) server_negotiation.cc:407] Connection header received
0929 16:34:37.168299 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.168305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.168415 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.168542 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.169587 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41286: BlockingRecv error: recv got EOF from 127.0.0.1:41286 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 16:34:37.245529 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.243120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.5.232.189:44389)
0929 16:34:37.243318 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.243325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.243956 (+ 631us) server_negotiation.cc:407] Connection header received
0929 16:34:37.244155 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.244161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.244255 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.244368 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.245256 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41290: BlockingRecv error: recv got EOF from 127.0.0.1:41290 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 16:34:37.379742 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.377557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.5.232.189:44389)
0929 16:34:37.377757 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.377762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.378181 (+ 419us) server_negotiation.cc:407] Connection header received
0929 16:34:37.378393 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.378398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.378487 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.378590 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.379537 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41304: BlockingRecv error: recv got EOF from 127.0.0.1:41304 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:34:37.645362 28570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:37.643278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.5.232.189:44389)
0929 16:34:37.643515 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:37.643522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:37.643928 (+ 406us) server_negotiation.cc:407] Connection header received
0929 16:34:37.644168 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:37.644172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:37.644274 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:37.644394 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:37.645193 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41310: BlockingRecv error: recv got EOF from 127.0.0.1:41310 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 16:34:37.692273 28662 catalog_manager.cc:1259] Loaded cluster ID: 4a13a0062cfd47fc8c9b9722bf85b46c
I20260929 16:34:37.692387 28662 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada: loading cluster ID for follower catalog manager: success
I20260929 16:34:37.694944 28662 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada: acquiring CA information for follower catalog manager: success
I20260929 16:34:37.696815 28662 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d4085ec065ce4ac5a4d0f406a3d61ada: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:34:37.926110 28708 catalog_manager.cc:1259] Loaded cluster ID: 4a13a0062cfd47fc8c9b9722bf85b46c
I20260929 16:34:37.926261 28708 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4: loading cluster ID for follower catalog manager: success
I20260929 16:34:37.928884 28708 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4: acquiring CA information for follower catalog manager: success
I20260929 16:34:37.931058 28708 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71c3df175d2e4cfb9bb2f8e7c42d38a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:34:38.171434 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:38.163515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.5.232.189:44389)
0929 16:34:38.164235 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:38.164242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:38.164540 (+ 298us) server_negotiation.cc:407] Connection header received
0929 16:34:38.164795 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:38.164799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:38.164952 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:38.165066 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:38.165670 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.166592 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.168643 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.169145 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.169245 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:38.169982 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:38.170660 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:38.170790 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:38.170936 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 16:34:38.171010 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":354,"threads_started":1}
I20260929 16:34:38.189743 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:38.182403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.5.232.188:33425)
0929 16:34:38.183222 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:38.183226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:38.183245 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:34:38.183351 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:38.183355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:38.183513 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:38.183668 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:38.184260 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.185014 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.187087 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.187523 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.187632 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:38.188349 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:38.188967 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:38.189094 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:38.189250 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:34:38.189325 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":604,"threads_started":1}
I20260929 16:34:38.207861 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:38.199578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.5.232.188:33425)
0929 16:34:38.199784 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:38.199792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:38.201105 (+ 1313us) server_negotiation.cc:407] Connection header received
0929 16:34:38.201399 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:38.201404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:38.201512 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:38.201578 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:38.202327 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.203417 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.205525 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.206158 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.206270 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:38.206988 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:38.207142 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:38.207211 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:38.207446 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 16:34:38.207537 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:34:38.215308 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:38.200571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.5.232.189:44389)
0929 16:34:38.200779 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:38.200785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:38.208836 (+ 8051us) server_negotiation.cc:407] Connection header received
0929 16:34:38.209122 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:38.209128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:38.209223 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:38.209291 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:38.209977 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.211087 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.213157 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.213843 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.213988 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:38.214456 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:38.214604 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:38.214709 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:38.214893 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 16:34:38.214985 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:34:38.216290 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:38.199644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.5.232.190:44059)
0929 16:34:38.209777 (+ 10133us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:38.209783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:38.209802 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:34:38.209929 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:38.209937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:38.210247 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:38.210454 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:38.211048 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.211949 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.213705 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:38.214367 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:38.214512 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:38.215075 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:38.215586 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:38.215752 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:38.215901 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:34:38.215990 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9983,"thread_start_us":9879,"threads_started":1}
I20260929 16:34:38.222182 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28425
I20260929 16:34:38.223311 28487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:38.360922 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 28425
W20260929 16:34:38.382269 28592 connection.cc:535] server connection from 127.0.0.1:53526 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:34:38.382696 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28499
I20260929 16:34:38.383780 28561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:38.493453 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 28499
I20260929 16:34:38.514127 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28573
I20260929 16:34:38.515262 28635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:38.626530 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 28573
2026-09-29T16:34:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5004 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:34:38 dist-test-slave-dssx krb5kdc[28794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:34:38 dist-test-slave-dssx krb5kdc[28794](info): set up 2 sockets
Sep 29 16:34:38 dist-test-slave-dssx krb5kdc[28794](info): commencing operation
krb5kdc: starting...
W20260929 16:34:39.750240 6050 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/test-user.keytab.
Sep 29 16:34:39 dist-test-slave-dssx krb5kdc[28794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.232.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
Entry for principal kudu/127.5.232.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab.
I20260929 16:34:40.019985 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46733
--webserver_interface=127.5.232.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:46733,127.5.232.189:39017,127.5.232.188:40927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:34:40.357764 28807 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:34:40.358173 28807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:40.358309 28807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:40.358461 28807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:40.366199 28807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:40.366334 28807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:40.366468 28807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:40.366559 28807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:40.366672 28807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:40.376833 28807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:46733,127.5.232.189:39017,127.5.232.188:40927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.190.keytab
--principal=kudu/127.5.232.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.190:46733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.232.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:40.378265 28807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:40.379633 28807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:40.391749 28813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:40.391700 28815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:40.391778 28812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:40.392606 28807 server_base.cc:1034] running on GCE node
I20260929 16:34:40.393489 28807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:40.395000 28807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:40.396260 28807 hybrid_clock.cc:648] HybridClock initialized: now 1790699680396194 us; error 39 us; skew 500 ppm
I20260929 16:34:40.396601 28807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:40 dist-test-slave-dssx krb5kdc[28794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:40.399859 28807 init.cc:377] Logged in from keytab as kudu/127.5.232.190@KRBTEST.COM (short username kudu)
I20260929 16:34:40.401786 28807 webserver.cc:460] Webserver started at http://127.5.232.190:40849/ using document root <none> and password file <none>
I20260929 16:34:40.402468 28807 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:40.402554 28807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:40.402828 28807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:40.405922 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/data/instance:
uuid: "ac94f7c96b364738ba99a3213d005d3b"
format_stamp: "Formatted at 2026-09-29 16:34:40 on dist-test-slave-dssx"
server_key: "7b9db1d2c67f45510db0a91647045c70"
server_key_iv: "c530a107623ba4dd16bdcb1dc1007903"
server_key_version: "encryptionkey@0"
I20260929 16:34:40.406698 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/wal/instance:
uuid: "ac94f7c96b364738ba99a3213d005d3b"
format_stamp: "Formatted at 2026-09-29 16:34:40 on dist-test-slave-dssx"
server_key: "7b9db1d2c67f45510db0a91647045c70"
server_key_iv: "c530a107623ba4dd16bdcb1dc1007903"
server_key_version: "encryptionkey@0"
I20260929 16:34:40.412139 28807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 16:34:40.415750 28822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:40.417186 28807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:34:40.417374 28807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/wal
uuid: "ac94f7c96b364738ba99a3213d005d3b"
format_stamp: "Formatted at 2026-09-29 16:34:40 on dist-test-slave-dssx"
server_key: "7b9db1d2c67f45510db0a91647045c70"
server_key_iv: "c530a107623ba4dd16bdcb1dc1007903"
server_key_version: "encryptionkey@0"
I20260929 16:34:40.417626 28807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:40.433665 28807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:40.437795 28807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:40.438097 28807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:40.456311 28807 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.190:46733
I20260929 16:34:40.456333 28873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.190:46733 every 8 connection(s)
I20260929 16:34:40.457841 28807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/data/info.pb
I20260929 16:34:40.461956 28874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:40.466142 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 28807
I20260929 16:34:40.466341 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-0/wal/instance
I20260929 16:34:40.466799 6050 external_mini_cluster.cc:1428] Setting key 51b79bf8ec556f7b279a833c6d2e765a
I20260929 16:34:40.468930 28874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:40 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.189@KRBTEST.COM; defaulting to no policy
I20260929 16:34:40.492504 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.472413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46733 (local address 127.0.0.1:41162)
0929 16:34:40.473363 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.473451 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:40.475666 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:40.476565 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:40.476571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:40.477297 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:40.477716 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:40.477735 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:40.479330 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:40.479339 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:40.480410 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:40.480430 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:40.481399 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:40.481407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:40.481609 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:40.482317 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:40.482345 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:40.484811 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:40.488135 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:40.488149 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:40.488163 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:40.488567 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:40.488982 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:40.488988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:40.488992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:40.489160 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:40.489995 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:40.490009 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:40.490238 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 16:34:40.490489 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 16:34:40.491098 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":148,"thread_start_us":364,"threads_started":1}
I20260929 16:34:40.492528 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.473816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41162 (local address 127.5.232.190:46733)
0929 16:34:40.474529 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:40.474542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:40.474595 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:34:40.476079 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:40.476085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:40.476337 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:40.476553 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:40.477940 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:40.479154 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:40.480564 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:40.481135 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:40.481353 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:40.485044 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:40.485052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:40.485067 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:40.485100 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:40.487966 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:40.488718 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:40.488725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:40.488727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:40.488870 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:40.489274 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:40.489279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:40.489288 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:40.489859 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:40.490086 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:40.490623 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 16:34:40.490980 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":506,"thread_start_us":298,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.189@KRBTEST.COM" created.
I20260929 16:34:40.496161 28874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:34:40.497265 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.496859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50772)
0929 16:34:40.497022 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.497128 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 16:34:40.497532 28824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
I20260929 16:34:40.499109 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.498730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50780)
0929 16:34:40.498915 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.498982 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:34:40.499290 28874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39017: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab.
Entry for principal kudu/127.5.232.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab.
I20260929 16:34:40.525116 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.189:39017
--webserver_interface=127.5.232.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab
--principal=kudu/127.5.232.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:46733,127.5.232.189:39017,127.5.232.188:40927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:34:40.548640 28874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } attempt: 1
I20260929 16:34:40.549992 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.549515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50794)
0929 16:34:40.549728 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.549845 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 16:34:40.551062 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.550730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50800)
0929 16:34:40.550865 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.550936 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:34:40.551326 28874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39017: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
I20260929 16:34:40.619561 28874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } attempt: 2
I20260929 16:34:40.620850 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.620366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50806)
0929 16:34:40.620607 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.620699 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 16:34:40.621960 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.621568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50818)
0929 16:34:40.621748 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.621831 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:34:40.622195 28874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39017: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
I20260929 16:34:40.713480 28874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } attempt: 3
I20260929 16:34:40.714697 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.714230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50822)
0929 16:34:40.714455 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.714555 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:34:40.715907 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.715483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50828)
0929 16:34:40.715672 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.715751 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:34:40.716099 28874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39017: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
I20260929 16:34:40.859408 28874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } attempt: 4
I20260929 16:34:40.860658 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.860106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50832)
0929 16:34:40.860418 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.860514 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 16:34:40.861795 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.861317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50836)
0929 16:34:40.861527 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.861660 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260929 16:34:40.862051 28874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.189:39017: Network error: Client connection negotiation failed: client connection to 127.5.232.189:39017: connect: Connection refused (error 111)
W20260929 16:34:40.871922 28881 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:34:40.872318 28881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:40.872431 28881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:40.872581 28881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:34:40.880367 28881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:40.880470 28881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:40.880574 28881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:40.880661 28881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:40.880782 28881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:40.891450 28881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:46733,127.5.232.189:39017,127.5.232.188:40927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.189.keytab
--principal=kudu/127.5.232.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.189:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.232.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:40.892828 28881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:40.894234 28881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:40.906152 28886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:40.906095 28887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:40.906188 28889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:40.907269 28881 server_base.cc:1034] running on GCE node
I20260929 16:34:40.908073 28881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:40.909524 28881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:40.910737 28881 hybrid_clock.cc:648] HybridClock initialized: now 1790699680910660 us; error 50 us; skew 500 ppm
I20260929 16:34:40.911069 28881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:40 dist-test-slave-dssx krb5kdc[28794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:40.914275 28881 init.cc:377] Logged in from keytab as kudu/127.5.232.189@KRBTEST.COM (short username kudu)
I20260929 16:34:40.915822 28881 webserver.cc:460] Webserver started at http://127.5.232.189:33531/ using document root <none> and password file <none>
I20260929 16:34:40.916515 28881 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:40.916643 28881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:40.916954 28881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:40.919945 28881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/data/instance:
uuid: "483edd7976fa444ba96b84a0acc320a1"
format_stamp: "Formatted at 2026-09-29 16:34:40 on dist-test-slave-dssx"
server_key: "a388469e9279a0f6a122ff7d4529dc08"
server_key_iv: "f23d6ede510f25d2893a7fbb0e490efa"
server_key_version: "encryptionkey@0"
I20260929 16:34:40.920766 28881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/wal/instance:
uuid: "483edd7976fa444ba96b84a0acc320a1"
format_stamp: "Formatted at 2026-09-29 16:34:40 on dist-test-slave-dssx"
server_key: "a388469e9279a0f6a122ff7d4529dc08"
server_key_iv: "f23d6ede510f25d2893a7fbb0e490efa"
server_key_version: "encryptionkey@0"
I20260929 16:34:40.926141 28881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 16:34:40.929710 28896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:40.931141 28881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:34:40.931331 28881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/wal
uuid: "483edd7976fa444ba96b84a0acc320a1"
format_stamp: "Formatted at 2026-09-29 16:34:40 on dist-test-slave-dssx"
server_key: "a388469e9279a0f6a122ff7d4529dc08"
server_key_iv: "f23d6ede510f25d2893a7fbb0e490efa"
server_key_version: "encryptionkey@0"
I20260929 16:34:40.931561 28881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:40.948846 28881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:40.953543 28881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:40.953836 28881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:40.972946 28881 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.189:39017
I20260929 16:34:40.972954 28947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.189:39017 every 8 connection(s)
I20260929 16:34:40.974423 28881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/data/info.pb
I20260929 16:34:40.978411 28948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:40.981793 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 28881
I20260929 16:34:40.981993 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-1/wal/instance
I20260929 16:34:40.982434 6050 external_mini_cluster.cc:1428] Setting key 89a26cb4b8538adc8b08d5576f03f622
I20260929 16:34:40.985270 28948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:40 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.232.188@KRBTEST.COM; defaulting to no policy
I20260929 16:34:41.008745 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.988237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.5.232.190:46733)
0929 16:34:40.988481 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:40.988489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:40.989853 (+ 1364us) server_negotiation.cc:407] Connection header received
0929 16:34:40.991898 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:40.991907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:40.992006 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:40.992304 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:40.993947 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:40.995094 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:40.996706 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:40.997536 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:40.997697 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.001838 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.001844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.001848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.001886 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.005026 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.006230 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.006234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.006237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.006364 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.006920 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.006924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.006926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.007285 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.007413 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.008318 (+ 905us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.008507 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.232.188@KRBTEST.COM" created.
I20260929 16:34:41.010069 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:40.988747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46733 (local address 127.0.0.1:41174)
0929 16:34:40.989636 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:40.989697 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:40.991582 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:40.992505 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:40.992515 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:40.993066 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:40.993759 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:40.993778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:40.995271 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:40.995276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:40.996523 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:40.996536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:40.997750 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:40.997755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:40.997968 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:40.998716 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:40.998740 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:41.001505 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:41.005241 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.005253 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.005275 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.006082 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.006493 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.006499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.006502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.006796 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.007427 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:41.007439 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:41.007844 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 16:34:41.008173 (+ 329us) client_negotiation.cc:240] Negotiation successful
0929 16:34:41.008596 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":322,"threads_started":1}
I20260929 16:34:41.011901 28948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:34:41.027549 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.012951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.5.232.189:39017)
0929 16:34:41.013705 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.013713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.013741 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:34:41.013790 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.013796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.013973 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.014150 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.014921 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.015985 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.017152 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.017783 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.017924 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.021457 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.021471 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.021486 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.021516 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.024375 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.025266 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.025273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.025275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.025445 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.025885 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.025889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.025891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.026263 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.026416 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.026879 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.027236 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":606,"thread_start_us":411,"threads_started":1}
I20260929 16:34:41.027637 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.012671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50838)
0929 16:34:41.012889 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.012905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:41.012995 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:41.014128 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:41.014132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:41.014596 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:41.014735 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.014743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.016155 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.016165 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.017009 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.017019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.017933 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.017938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.018131 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.018783 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:41.018800 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:41.021145 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:41.024533 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.024541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.024548 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.025126 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.025574 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.025588 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.025594 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.025782 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.026414 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:41.026422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:41.026607 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 16:34:41.026762 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 16:34:41.027107 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:34:41.030398 28948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:34:41.031661 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.031049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53536)
0929 16:34:41.031267 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.031426 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 16:34:41.031944 28898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
I20260929 16:34:41.033779 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.033228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53546)
0929 16:34:41.033483 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.033605 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 16:34:41.034037 28948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40927: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.232.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab.
Entry for principal kudu/127.5.232.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab.
I20260929 16:34:41.044449 6050 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_4zdm2/build/asan/bin/kudu
/tmp/dist-test-task_4zdm2/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.188:40927
--webserver_interface=127.5.232.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab
--principal=kudu/127.5.232.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.232.148:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.232.190:46733,127.5.232.189:39017,127.5.232.188:40927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:34:41.083410 28948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } attempt: 1
I20260929 16:34:41.084724 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.084142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53548)
0929 16:34:41.084391 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.084530 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 16:34:41.085984 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.085567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53558)
0929 16:34:41.085745 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.085844 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:34:41.086257 28948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40927: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
I20260929 16:34:41.154513 28948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } attempt: 2
I20260929 16:34:41.155887 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.155273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53562)
0929 16:34:41.155554 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.155686 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 16:34:41.157058 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.156612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53568)
0929 16:34:41.156812 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.156910 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 16:34:41.157302 28948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40927: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
I20260929 16:34:41.161361 28874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } attempt: 5
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:34:41.175933 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.162168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50840)
0929 16:34:41.162382 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.162404 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:41.162535 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:41.162903 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:41.162906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:41.163281 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:41.163481 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.163489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.164927 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.164935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.165894 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.165908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.166940 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.166944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.167105 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.167767 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:41.167784 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:41.169896 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:41.173616 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.173621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.173624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.174029 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.174441 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.174444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.174446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.174574 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.175180 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:41.175184 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:41.175338 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 16:34:41.175460 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:34:41.175679 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:34:41.176352 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.162364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.5.232.189:39017)
0929 16:34:41.162553 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.162559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.162582 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:34:41.162667 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.162672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.162755 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.162870 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.163647 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.164747 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.166055 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.166786 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.166927 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.170138 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.170146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.170150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.170200 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.173459 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.174166 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.174172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.174175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.174328 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.174677 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.174682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.174685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.175059 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.175220 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.175632 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.175920 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:34:41.177564 28874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:34:41.178592 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.178183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53578)
0929 16:34:41.178355 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.178444 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:34:41.179502 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.179179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53592)
0929 16:34:41.179318 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.179403 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:34:41.179703 28874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40927: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
I20260929 16:34:41.230927 28874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } attempt: 1
I20260929 16:34:41.232278 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.231809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53604)
0929 16:34:41.232011 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.232107 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:34:41.233297 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.232960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53620)
0929 16:34:41.233118 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.233186 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:34:41.233537 28874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40927: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
I20260929 16:34:41.248566 28948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } attempt: 3
I20260929 16:34:41.249964 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.249240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53634)
0929 16:34:41.249571 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.249724 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 16:34:41.251385 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.250749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53638)
0929 16:34:41.251054 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.251191 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260929 16:34:41.251662 28948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40927: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
I20260929 16:34:41.301750 28874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } attempt: 2
I20260929 16:34:41.302994 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.302480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53652)
0929 16:34:41.302722 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.302823 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 16:34:41.304057 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.303692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53658)
0929 16:34:41.303851 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.303948 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:34:41.304306 28874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40927: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
I20260929 16:34:41.394915 28948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } attempt: 4
W20260929 16:34:41.394912 28955 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:34:41.395344 28955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:34:41.395468 28955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:34:41.395627 28955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:34:41.396238 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.395664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53660)
0929 16:34:41.395921 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.396030 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 16:34:41.397269 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.396898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53676)
0929 16:34:41.397047 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.397132 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:34:41.397503 28948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40927: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
W20260929 16:34:41.403861 28955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:34:41.403960 28955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:34:41.404062 28955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:34:41.404152 28955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:34:41.404241 28955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:34:41.410531 28874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } attempt: 3
I20260929 16:34:41.411669 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.411174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53678)
0929 16:34:41.411430 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.411514 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 16:34:41.412695 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.412331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53684)
0929 16:34:41.412517 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.412590 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 16:34:41.412911 28874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.232.188:40927: Network error: Client connection negotiation failed: client connection to 127.5.232.188:40927: connect: Connection refused (error 111)
I20260929 16:34:41.415639 28955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.232.148:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.232.190:46733,127.5.232.189:39017,127.5.232.188:40927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/krb5kdc/kudu_127.5.232.188.keytab
--principal=kudu/127.5.232.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.232.188:40927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.232.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:34:41.417037 28955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:34:41.418502 28955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:34:41.431030 28960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:41.431084 28961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:34:41.431260 28963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:34:41.431963 28955 server_base.cc:1034] running on GCE node
I20260929 16:34:41.432775 28955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:34:41.434350 28955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:34:41.435598 28955 hybrid_clock.cc:648] HybridClock initialized: now 1790699681435513 us; error 61 us; skew 500 ppm
I20260929 16:34:41.435961 28955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699681, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:34:41.439112 28955 init.cc:377] Logged in from keytab as kudu/127.5.232.188@KRBTEST.COM (short username kudu)
I20260929 16:34:41.440744 28955 webserver.cc:460] Webserver started at http://127.5.232.188:38135/ using document root <none> and password file <none>
I20260929 16:34:41.441530 28955 fs_manager.cc:359] Metadata directory not provided
I20260929 16:34:41.441648 28955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:34:41.442008 28955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:34:41.445147 28955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/data/instance:
uuid: "68d0c1efab0d4c05a384489ad63b8a70"
format_stamp: "Formatted at 2026-09-29 16:34:41 on dist-test-slave-dssx"
server_key: "d6e14d0dfa360cef7ab9c259e23bdc1d"
server_key_iv: "191a9bfd82003a1f7fc5879af20f8266"
server_key_version: "encryptionkey@0"
I20260929 16:34:41.446118 28955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/wal/instance:
uuid: "68d0c1efab0d4c05a384489ad63b8a70"
format_stamp: "Formatted at 2026-09-29 16:34:41 on dist-test-slave-dssx"
server_key: "d6e14d0dfa360cef7ab9c259e23bdc1d"
server_key_iv: "191a9bfd82003a1f7fc5879af20f8266"
server_key_version: "encryptionkey@0"
I20260929 16:34:41.451683 28955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:34:41.455358 28970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:34:41.457181 28955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:34:41.457365 28955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/data,/tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/wal
uuid: "68d0c1efab0d4c05a384489ad63b8a70"
format_stamp: "Formatted at 2026-09-29 16:34:41 on dist-test-slave-dssx"
server_key: "d6e14d0dfa360cef7ab9c259e23bdc1d"
server_key_iv: "191a9bfd82003a1f7fc5879af20f8266"
server_key_version: "encryptionkey@0"
I20260929 16:34:41.457640 28955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:34:41.481896 28955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:34:41.486577 28955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:34:41.486856 28955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:34:41.506138 28955 rpc_server.cc:307] RPC server started. Bound to: 127.5.232.188:40927
I20260929 16:34:41.506173 29021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.232.188:40927 every 8 connection(s)
I20260929 16:34:41.507788 28955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/data/info.pb
I20260929 16:34:41.511703 6050 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_4zdm2/build/asan/bin/kudu as pid 28955
I20260929 16:34:41.511909 6050 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_4zdm2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699314866137-6050-0/minicluster-data/master-2/wal/instance
I20260929 16:34:41.512296 29022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:34:41.512497 6050 external_mini_cluster.cc:1428] Setting key fccb6727d01c26c55093e873c811f637
I20260929 16:34:41.521620 29022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699681, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
I20260929 16:34:41.541903 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.524275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.5.232.190:46733)
0929 16:34:41.525277 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.525284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.525305 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:34:41.525521 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.525526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.525606 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.525717 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.526839 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.527606 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.528895 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.529294 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.529457 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.534973 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.534983 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.534991 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.535038 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.538072 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.539051 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.539055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.539057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.539181 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.539627 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.539632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.539635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.540016 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.540170 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.541164 (+ 994us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.541440 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":322,"threads_started":1}
I20260929 16:34:41.543087 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.524335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.190:46733 (local address 127.0.0.1:41178)
0929 16:34:41.525187 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.525227 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:41.525356 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:41.525827 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:41.525833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:41.526459 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:41.526705 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.526713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.527748 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.527752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.528764 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.528774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.529557 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.529562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.529689 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.531751 (+ 2062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:41.531780 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:41.534705 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:41.538220 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.538230 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.538244 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.538920 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.539286 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.539291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.539294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.539516 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.540167 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:41.540176 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:41.540716 (+ 540us) client_negotiation.cc:769] Sending connection context
0929 16:34:41.541050 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 16:34:41.541598 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":478,"threads_started":1}
I20260929 16:34:41.543111 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.514486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.5.232.188:40927)
0929 16:34:41.515297 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.515309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.515360 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:34:41.517366 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.517451 (+ 85us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.517944 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.518422 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.519936 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.521467 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.523074 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.523866 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.524064 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.535547 (+ 11483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.535556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.535569 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.535602 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.539281 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.540144 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.540155 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.540157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.540317 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.540678 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.540683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.540686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.541077 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.541357 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.541767 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.542006 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":375,"threads_started":1}
I20260929 16:34:41.544847 29022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699681, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.190@KRBTEST.COM
I20260929 16:34:41.559772 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.545568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.189:39017 (local address 127.0.0.1:50856)
0929 16:34:41.545760 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.545778 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:41.545876 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:41.546215 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:41.546218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:41.546677 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:41.546905 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.546918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.548351 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.548355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.549689 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.549704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.550715 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.550719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.550874 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.551523 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:41.551545 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:41.553940 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:41.557198 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.557204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.557208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.557696 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.558091 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.558096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.558100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.558244 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.558777 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:41.558782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:41.558949 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 16:34:41.559085 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 16:34:41.559377 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:34:41.559793 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.545702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.5.232.189:39017)
0929 16:34:41.545884 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.545888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.545905 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:34:41.545984 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.545988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.546081 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.546194 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.547069 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.548194 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.549880 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.550568 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.550705 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.554167 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.554176 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.554183 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.554233 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.557041 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.557822 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.557828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.557832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.557975 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.558362 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.558366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.558372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.558659 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.558817 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.559200 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.559415 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:34:41.560818 29022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:34:41.562855 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.546808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.5.232.190:46733)
0929 16:34:41.547010 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.547016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.547033 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:34:41.547079 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.547091 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.547157 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.547227 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.548711 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.549748 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.551535 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.552205 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.552347 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.556930 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.556936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.556940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.556976 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.559653 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.560533 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.560540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.560546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.560731 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.561182 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.561188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.561191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.561631 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.562079 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.562238 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.562507 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699681, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.188@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.232.189@KRBTEST.COM
I20260929 16:34:41.577425 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.561817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.5.232.188:40927)
0929 16:34:41.561955 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.561960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.561976 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:34:41.562112 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.562120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.562204 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.562302 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.563189 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.564140 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.565819 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.566208 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.566309 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.570026 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.570033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.570038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.570081 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.573817 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.574759 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.574766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.574770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.574972 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.575538 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.575545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.575549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.575982 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.576153 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.576690 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.576993 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:34:41.577476 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.561539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53700)
0929 16:34:41.561856 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.561877 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:41.561972 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:41.562346 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:41.562351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:41.562861 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:41.563054 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.563069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.564311 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.564317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.565664 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.565680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.566359 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.566365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.566605 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.567267 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:41.567284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:41.569773 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:41.574018 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.574027 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.574032 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.574622 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.575153 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.575159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.575163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.575343 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.576135 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:41.576140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:41.576329 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 16:34:41.576501 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 16:34:41.577134 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260929 16:34:41.582650 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.564640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.5.232.189:39017)
0929 16:34:41.564810 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.564814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.564848 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:34:41.565029 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.565033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.565090 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.565167 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.566725 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.567627 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.569224 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.569901 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.570013 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.575813 (+ 5800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.575822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.575826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.575895 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.579530 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.580453 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.580458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.580460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.580608 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.581089 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.581094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.581098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.581463 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.581709 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.582112 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.582307 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:34:41.583539 29022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70: Bootstrap starting.
I20260929 16:34:41.587635 29022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:41.587772 6050 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:34:41.588898 29022 log.cc:824] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:41.590155 28874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } attempt: 4
I20260929 16:34:41.592393 29022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70: No bootstrap required, opened a new log
I20260929 16:34:41.590406 28849 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ac94f7c96b364738ba99a3213d005d3b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41192
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.190@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
I20260929 16:34:41.602279 29022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } }
I20260929 16:34:41.602795 29022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d0c1efab0d4c05a384489ad63b8a70, State: Initialized, Role: FOLLOWER
I20260929 16:34:41.603390 29022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } }
I20260929 16:34:41.604543 29029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.605026 29029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:41.605770 29022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:41.607327 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.590857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.5.232.188:40927)
0929 16:34:41.591051 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.591055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.591074 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:34:41.591311 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.591317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.591397 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.592575 (+ 1178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.592732 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.593894 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.595713 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.596452 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.596592 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.600162 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.600169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.600175 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.600230 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.603923 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.604825 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.604831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.604835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.604999 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.605532 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.605537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.605540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.605988 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.606158 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.606649 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.606873 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:34:41.607348 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.590794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53714)
0929 16:34:41.591005 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.591025 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:41.591144 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:41.591563 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:41.591568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:41.592172 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:41.592414 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.592427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.594088 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.594093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.595535 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.595551 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.596616 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.596631 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.596836 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.597581 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:41.597600 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:41.599937 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:41.604099 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.604106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.604111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.604680 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.605161 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.605167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.605170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.605357 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.606121 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:41.606128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:41.606362 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 16:34:41.606527 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 16:34:41.606840 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:34:41.612660 28874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b: Bootstrap starting.
I20260929 16:34:41.615896 28874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:41.617278 28874 log.cc:824] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:41.621506 28874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b: No bootstrap required, opened a new log
W20260929 16:34:41.624670 29041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:34:41.624800 29041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:34:41.629526 28874 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } }
I20260929 16:34:41.629973 28874 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac94f7c96b364738ba99a3213d005d3b, State: Initialized, Role: FOLLOWER
I20260929 16:34:41.630561 28874 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } }
I20260929 16:34:41.631863 29043 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.632555 29043 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:41.633083 28874 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:41.633183 28849 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:34:41.633414 28849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:41.622186 28955 master_runner.cc:186] Error getting master registration for 127.5.232.190:46733: OK, instance_id { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" instance_seqno: 1790699680451267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:34:41.633795 28955 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:34:41.636257 28849 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } }
I20260929 16:34:41.638542 28997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac94f7c96b364738ba99a3213d005d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "68d0c1efab0d4c05a384489ad63b8a70"
I20260929 16:34:41.639088 28997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:41.641968 28849 leader_election.cc:288] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [CANDIDATE]: Term 1 election: Requested vote from peers 483edd7976fa444ba96b84a0acc320a1 (127.5.232.189:39017), 68d0c1efab0d4c05a384489ad63b8a70 (127.5.232.188:40927)
I20260929 16:34:41.642719 28997 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac94f7c96b364738ba99a3213d005d3b in term 1.
I20260929 16:34:41.643396 28824 leader_election.cc:302] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68d0c1efab0d4c05a384489ad63b8a70, ac94f7c96b364738ba99a3213d005d3b; no voters:
I20260929 16:34:41.643989 29043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:34:41.648909 29043 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [term 1 LEADER]: Becoming Leader. State: Replica: ac94f7c96b364738ba99a3213d005d3b, State: Running, Role: LEADER
I20260929 16:34:41.649741 29043 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } }
I20260929 16:34:41.658663 29047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac94f7c96b364738ba99a3213d005d3b. Latest consensus state: current_term: 1 leader_uuid: "ac94f7c96b364738ba99a3213d005d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.659160 29047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:41.678443 29055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:34:41.685173 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.656918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.5.232.189:39017)
0929 16:34:41.657099 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.657106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.658564 (+ 1458us) server_negotiation.cc:407] Connection header received
0929 16:34:41.659492 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.659498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.659583 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.659675 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.663607 (+ 3932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.664781 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.668092 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.668982 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.669122 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.675287 (+ 6165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.675296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.675301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.675354 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.679455 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.681024 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.681031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.681035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.681202 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.682340 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.682346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.682349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.682758 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.682997 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.684489 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.684772 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:34:41.691010 28828 tablet.cc:2369] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:41.696688 28948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } attempt: 5
I20260929 16:34:41.708724 29055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:34:41.711994 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.655580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.5.232.190:46733)
0929 16:34:41.655821 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.655827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.656997 (+ 1170us) server_negotiation.cc:407] Connection header received
0929 16:34:41.657452 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.657458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.657534 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.657610 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.663610 (+ 6000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.664774 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.673682 (+ 8908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.674449 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.674571 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.699473 (+ 24902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.699486 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.699495 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.699559 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.704579 (+ 5020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.706489 (+ 1910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.706495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.706498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.706853 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.707664 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.707705 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.707710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.708379 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.709021 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.710418 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.710903 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:34:41.719488 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.660019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.5.232.188:40927)
0929 16:34:41.660230 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.660238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.673331 (+ 13093us) server_negotiation.cc:407] Connection header received
0929 16:34:41.675736 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.675746 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.675844 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.675923 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.680746 (+ 4823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.681972 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.683941 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.684662 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.684766 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.709180 (+ 24414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.709187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.709202 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.709242 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.712916 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.713900 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.713905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.713908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.714081 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.717504 (+ 3423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.717512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.717517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.717911 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.718177 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.718840 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.719073 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:34:41.726091 29066 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Sep 29 16:34:41 dist-test-slave-dssx krb5kdc[28794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.232.189@KRBTEST.COM for kudu/127.5.232.188@KRBTEST.COM
I20260929 16:34:41.730235 28997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [term 1 FOLLOWER]: Refusing update from remote peer ac94f7c96b364738ba99a3213d005d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:41.733305 29043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:34:41.738973 28976 tablet.cc:2369] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:34:41.740389 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.697278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.232.188:40927 (local address 127.0.0.1:53734)
0929 16:34:41.697687 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 16:34:41.697707 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:34:41.697823 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:34:41.706839 (+ 9016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:34:41.706848 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:34:41.707583 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:34:41.707889 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.707908 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.710495 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.710503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.712393 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:34:41.712418 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.717348 (+ 4930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.717358 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:34:41.717707 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.718404 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:34:41.718429 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:34:41.727282 (+ 8853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:34:41.735495 (+ 8213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.735501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.735506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.736174 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.736743 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:34:41.736748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:34:41.736751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:34:41.736922 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:34:41.739294 (+ 2372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:34:41.739299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:34:41.739477 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 16:34:41.739632 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 16:34:41.739998 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260929 16:34:41.741796 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:41.698043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.5.232.188:40927)
0929 16:34:41.706261 (+ 8218us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:41.706271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:41.706309 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:34:41.706390 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:41.706399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:41.706520 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:41.708147 (+ 1627us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:41.708482 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.710270 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.712724 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:41.717124 (+ 4400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:41.718072 (+ 948us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:41.727752 (+ 9680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:34:41.727761 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:34:41.727767 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:34:41.727852 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:34:41.735309 (+ 7457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.736404 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.736409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.736413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.736596 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:34:41.737110 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:34:41.737115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:34:41.737118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:34:41.739141 (+ 2023us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:34:41.740946 (+ 1805us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:41.741119 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 16:34:41.741343 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7969,"thread_start_us":7728,"threads_started":1}
I20260929 16:34:41.748082 28948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1: Bootstrap starting.
I20260929 16:34:41.735260 28807 master_runner.cc:186] Error getting master registration for 127.5.232.189:39017: OK, instance_id { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" instance_seqno: 1790699680967527 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:34:41.752166 28807 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:34:41.753543 28948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1: Neither blocks nor log segments found. Creating new log.
I20260929 16:34:41.754910 28948 log.cc:824] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:34:41.758939 29029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac94f7c96b364738ba99a3213d005d3b. Latest consensus state: current_term: 1 leader_uuid: "ac94f7c96b364738ba99a3213d005d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.759418 29029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:41.764008 29043 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac94f7c96b364738ba99a3213d005d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.764351 29043 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:41.767980 29055 catalog_manager.cc:1347] Generated new cluster ID: 70df80df58d948adb2b1a2afb515152a
I20260929 16:34:41.768100 29055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:34:41.773922 29029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac94f7c96b364738ba99a3213d005d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.774231 29029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:41.797087 28948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1: No bootstrap required, opened a new log
I20260929 16:34:41.808279 28948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } }
I20260929 16:34:41.808776 28948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 483edd7976fa444ba96b84a0acc320a1, State: Initialized, Role: FOLLOWER
I20260929 16:34:41.809333 28948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } }
I20260929 16:34:41.816392 29073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.816874 29073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:41.818980 28948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:34:41.819455 28922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:34:41.819716 28923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac94f7c96b364738ba99a3213d005d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "483edd7976fa444ba96b84a0acc320a1"
I20260929 16:34:41.824753 28922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [term 1 FOLLOWER]: Refusing update from remote peer ac94f7c96b364738ba99a3213d005d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:34:41.830744 28923 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac94f7c96b364738ba99a3213d005d3b in term 1.
I20260929 16:34:41.836048 29043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:34:41.842748 29055 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:34:41.845551 29055 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:34:41.873790 29073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac94f7c96b364738ba99a3213d005d3b. Latest consensus state: current_term: 1 leader_uuid: "ac94f7c96b364738ba99a3213d005d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.874150 29073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:41.874660 29080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac94f7c96b364738ba99a3213d005d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.875108 29080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:34:41.888334 29047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac94f7c96b364738ba99a3213d005d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac94f7c96b364738ba99a3213d005d3b" member_type: VOTER last_known_addr { host: "127.5.232.190" port: 46733 } } peers { permanent_uuid: "483edd7976fa444ba96b84a0acc320a1" member_type: VOTER last_known_addr { host: "127.5.232.189" port: 39017 } } peers { permanent_uuid: "68d0c1efab0d4c05a384489ad63b8a70" member_type: VOTER last_known_addr { host: "127.5.232.188" port: 40927 } } }
I20260929 16:34:41.888731 29047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b [sys.catalog]: This master's current role is: LEADER
I20260929 16:34:41.897760 29055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac94f7c96b364738ba99a3213d005d3b: Generated new TSK 0
I20260929 16:34:41.899003 29055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:34:41.937073 29089 catalog_manager.cc:1259] Loaded cluster ID: 70df80df58d948adb2b1a2afb515152a
I20260929 16:34:41.937216 29089 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1: loading cluster ID for follower catalog manager: success
I20260929 16:34:41.941258 29089 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1: acquiring CA information for follower catalog manager: success
I20260929 16:34:41.955581 29089 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 483edd7976fa444ba96b84a0acc320a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:34:42.030715 29026 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.015430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.5.232.190:46733)
0929 16:34:42.015660 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.015667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.028457 (+ 12790us) server_negotiation.cc:407] Connection header received
0929 16:34:42.028984 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.028991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.029087 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.029204 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.030448 (+ 1244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41202: BlockingRecv error: recv got EOF from 127.0.0.1:41202 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 16:34:42.034768 28952 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.027736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.5.232.189:39017)
0929 16:34:42.027978 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.027985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.032301 (+ 4316us) server_negotiation.cc:407] Connection header received
0929 16:34:42.032985 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.032991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.033077 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.033168 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.034494 (+ 1326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50874: BlockingRecv error: recv got EOF from 127.0.0.1:50874 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 16:34:42.043080 29062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.040437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.5.232.188:40927)
0929 16:34:42.040654 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.040661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.041184 (+ 523us) server_negotiation.cc:407] Connection header received
0929 16:34:42.041488 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.041494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.041595 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.041669 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.042819 (+ 1150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53738: BlockingRecv error: recv got EOF from 127.0.0.1:53738 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:34:42.095782 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.077739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.5.232.189:39017)
0929 16:34:42.077965 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.077971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.087161 (+ 9190us) server_negotiation.cc:407] Connection header received
0929 16:34:42.087800 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.087807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.088055 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.088234 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.089034 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.090323 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.092544 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.093268 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.093403 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.094225 (+ 822us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.094953 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.095075 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.095331 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.095436 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:34:42.146978 29062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.135555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.5.232.188:40927)
0929 16:34:42.135760 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.135766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.144571 (+ 8805us) server_negotiation.cc:407] Connection header received
0929 16:34:42.145211 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.145218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.145366 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.145479 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.146693 (+ 1214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53746: BlockingRecv error: recv got EOF from 127.0.0.1:53746 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:34:42.201611 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.183319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.5.232.189:39017)
0929 16:34:42.183590 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.183599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.194266 (+ 10667us) server_negotiation.cc:407] Connection header received
0929 16:34:42.194884 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.194896 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.194998 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.195079 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.195804 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.196955 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.199170 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.199852 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.199967 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.200644 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.200822 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.200895 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.201154 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.201255 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 16:34:42.255625 29062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.242345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.5.232.188:40927)
0929 16:34:42.242580 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.242588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.253279 (+ 10691us) server_negotiation.cc:407] Connection header received
0929 16:34:42.253925 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.253932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.254036 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.254142 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.255360 (+ 1218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53754: BlockingRecv error: recv got EOF from 127.0.0.1:53754 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:34:42.325896 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.307432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.5.232.189:39017)
0929 16:34:42.307619 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.307626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.318137 (+ 10511us) server_negotiation.cc:407] Connection header received
0929 16:34:42.318425 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.318432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.318542 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.318622 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.319572 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.320864 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.323311 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.324039 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.324161 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.324886 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.325066 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.325140 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.325410 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.325512 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 16:34:42.373831 29062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.360465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.5.232.188:40927)
0929 16:34:42.360663 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.360669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.371504 (+ 10835us) server_negotiation.cc:407] Connection header received
0929 16:34:42.372135 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.372142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.372288 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.372377 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.373540 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53766: BlockingRecv error: recv got EOF from 127.0.0.1:53766 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:34:42.392824 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.383354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.5.232.189:39017)
0929 16:34:42.383566 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.383574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.386093 (+ 2519us) server_negotiation.cc:407] Connection header received
0929 16:34:42.386360 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.386367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.386489 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.386606 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.387154 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.388357 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.390450 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.391183 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.391291 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.391969 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.392113 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.392223 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.392402 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.392500 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 16:34:42.415973 29062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.406393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.5.232.188:40927)
0929 16:34:42.406670 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.406677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.414122 (+ 7445us) server_negotiation.cc:407] Connection header received
0929 16:34:42.414515 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.414522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.414629 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.414756 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.415734 (+ 978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53770: BlockingRecv error: recv got EOF from 127.0.0.1:53770 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20260929 16:34:42.436842 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.429438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.5.232.189:39017)
0929 16:34:42.429628 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.429635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.430211 (+ 576us) server_negotiation.cc:407] Connection header received
0929 16:34:42.430423 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.430429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.430527 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.430626 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.431090 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.432262 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.434374 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.435228 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.435357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.435954 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.436101 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.436191 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.436391 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.436482 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 16:34:42.445989 29062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.443443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.5.232.188:40927)
0929 16:34:42.443620 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.443626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.444188 (+ 562us) server_negotiation.cc:407] Connection header received
0929 16:34:42.444471 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.444480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.444586 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.444657 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.445761 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53778: BlockingRecv error: recv got EOF from 127.0.0.1:53778 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:34:42.477754 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.471355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.5.232.189:39017)
0929 16:34:42.471563 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.471570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.472086 (+ 516us) server_negotiation.cc:407] Connection header received
0929 16:34:42.472289 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.472293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.472366 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.472447 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.472902 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.473779 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.475720 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.476143 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.476251 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.476936 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.477055 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.477123 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.477296 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.477375 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:34:42.486301 29062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.484166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.5.232.188:40927)
0929 16:34:42.484409 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.484415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.484697 (+ 282us) server_negotiation.cc:407] Connection header received
0929 16:34:42.484878 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.484882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.484957 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.485040 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.486007 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53786: BlockingRecv error: recv got EOF from 127.0.0.1:53786 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 16:34:42.527334 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.521805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.5.232.189:39017)
0929 16:34:42.522046 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.522052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.522595 (+ 543us) server_negotiation.cc:407] Connection header received
0929 16:34:42.522814 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.522818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.522887 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.522971 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.523403 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.524093 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.525740 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.526199 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.526296 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.526633 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.526733 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.526806 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.526925 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.526991 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 16:34:42.534982 29062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.533097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.5.232.188:40927)
0929 16:34:42.533278 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.533282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.533730 (+ 448us) server_negotiation.cc:407] Connection header received
0929 16:34:42.533883 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.533888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.533965 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.534057 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.534816 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53800: BlockingRecv error: recv got EOF from 127.0.0.1:53800 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:34:42.607824 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.602726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.5.232.189:39017)
0929 16:34:42.602936 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.602941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.603267 (+ 326us) server_negotiation.cc:407] Connection header received
0929 16:34:42.603445 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.603449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.603534 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.603627 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.604013 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.604825 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.606277 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.606771 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.606872 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.607239 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.607352 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.607421 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.607553 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.607620 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 16:34:42.615208 29062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:34:42.613596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.5.232.188:40927)
0929 16:34:42.613776 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.613781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.614071 (+ 290us) server_negotiation.cc:407] Connection header received
0929 16:34:42.614221 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.614225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.614297 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.614392 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:34:42.615040 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53816: BlockingRecv error: recv got EOF from 127.0.0.1:53816 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:34:42.626853 29041 catalog_manager.cc:1259] Loaded cluster ID: 70df80df58d948adb2b1a2afb515152a
I20260929 16:34:42.626951 29041 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70: loading cluster ID for follower catalog manager: success
I20260929 16:34:42.629743 29041 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70: acquiring CA information for follower catalog manager: success
I20260929 16:34:42.631644 29041 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68d0c1efab0d4c05a384489ad63b8a70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:34:42.752157 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.746833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.5.232.189:39017)
0929 16:34:42.747043 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.747047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.747479 (+ 432us) server_negotiation.cc:407] Connection header received
0929 16:34:42.747654 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.747658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.747734 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.747832 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.748313 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.749019 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.750430 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.750852 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.750946 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.751359 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.751475 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.751544 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.751846 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.751924 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:34:42.764497 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.758546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.5.232.188:40927)
0929 16:34:42.758784 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.758791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.759074 (+ 283us) server_negotiation.cc:407] Connection header received
0929 16:34:42.759215 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.759219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.759378 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.759482 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.759864 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.760611 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.762059 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.762498 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.762603 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.763258 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.763843 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.763956 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.764036 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.764148 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:34:42.778944 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.771630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.5.232.189:39017)
0929 16:34:42.771820 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.771826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.772294 (+ 468us) server_negotiation.cc:407] Connection header received
0929 16:34:42.772649 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.772658 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.772772 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.772840 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.773681 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.774610 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.776865 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.777459 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.777596 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.778126 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.778252 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.778352 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.778562 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.778665 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:34:42.779089 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.772431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.5.232.188:40927)
0929 16:34:42.772605 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.772612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.773237 (+ 625us) server_negotiation.cc:407] Connection header received
0929 16:34:42.773433 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.773440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.773547 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.773646 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.774207 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.775079 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.776910 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.777622 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.777725 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.778319 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.778443 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.778553 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.778717 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.778801 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:34:42.779343 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:34:42.770759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.5.232.190:46733)
0929 16:34:42.771812 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0929 16:34:42.771819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:34:42.771841 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:34:42.771955 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:34:42.771959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:34:42.772114 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:34:42.772987 (+ 873us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:34:42.773169 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.774300 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.776508 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:34:42.777167 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:34:42.777303 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:34:42.777977 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:34:42.778474 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:34:42.778631 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:34:42.778736 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:34:42.778833 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":829,"threads_started":1}
I20260929 16:34:42.783869 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28807
I20260929 16:34:42.784952 28869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:42.897117 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 28807
I20260929 16:34:42.919312 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28881
I20260929 16:34:42.920557 28943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:43.029054 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 28881
I20260929 16:34:43.049520 6050 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_4zdm2/build/asan/bin/kudu pid 28955
I20260929 16:34:43.050639 29017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:34:43.158178 6050 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_4zdm2/build/asan/bin/kudu with pid 28955
2026-09-29T16:34:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4535 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9540 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (368254 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST